With the provocative title, “Publish-or-perish: Peer review and the corruption of science“, David Colquhoun lays out, what he believes to be some problems with the current status of peer review, highlighting a specific example involving a manuscript about acupuncture.
It seems to me that Colquhoun lament the emphasis on the quantity of publications, with respect to career promotions and grant applications, which drives two undesirable practices:
1. The parsing of papers to the LPU (least publishable unit), resulting in more publications, but without a comprehensive story, or with overstated results. Or, in extreme cases, outright fraud; and,
2. An over-burdening of the current review system, where there are not enough qualified reviewers to systematically, and carefully, review each manuscript.
This may lead to initiation up of more and more journals, which accommodate the publication of LPU papers, and may often suffer from a lack of qualified reviewers.
He pulls out a study from a peer-reviewed journal on acupuncture, which, in hind-sight, is not the best example. This paper doesn’t appear to have any glaring examples of fraud, it was not retracted, and responses from one of the paper’s authors and one of the journal’s editors, seem to contradict the claims made in his summary. Alternatively, Colquhoun could have cited one of the more highly-publicized examples of the scientific community questioning the peer-review process. The first one that comes to mind is the study of “arsenic-based life“, and the many researchers who voiced their concerns about it. Or, a recent paper on the genetics of longevity that was retracted due to technical differences in how the control and sample datasets were analyzed.
I do agree with Colquhoun that one beneficial alternative would be something similar to publishing your paper and leave it open to comments from the community. I don’t think that reviews need be anonymous, as he supposes, after citing a failure of such an endeavor at Nature. There is an example of a successful journal, Biology Direct, in which the peer review process is open, and the reviews (as well as author responses) are published alongside the manuscript. After a quick search, I found one really wonderful example, where I learned as much from the back-and-forth between the reviewers and the authors, as I did from the article. The conversation between the authors, and the reviewers, people who have been studying the evolution of life on earth, for decades was respectful, but with disagreements.
Maybe I’m optimistic, but I think there is a lot of benefit of public reviews. It is especially useful to read questions raised by experts that I, as someone unfamiliar with the field, would not know to ask.
Alternatively, I am not sure how to address the problem of quantity-over-quality in academic promotions and funding success. Colquhoun says it very well, I think, that, “It arises from official pressure to publish when you have nothing to say”. He also does not have many suggestions, and seems pessimistic about alternative measurements of scientific success.
Perhaps, rather than simply list publications, scientists could write a summary of the findings from their publications over the review period. This might equal the playing field between a manuscript that presents a comprehensive analysis, and a list of several manuscripts, each of which are short-stories. In addition, it would encourage the dissemination of such knowledge to the public (and encourage public support of science) because Institutions could share these short summaries online every time a new review is completed. A double win!
But, it seems like the current system is in place for the foreseeable future.
I wonder if blog posts can contribute to my publication list?