Let's review the peer review process
18 APRIL 2013
SOURCE: JAMES FRYER
We must hold up a mirror to scientific peer review if we are to stamp out fraud and uphold the discipline’s reputation, argues Philip Moriarty
The website Science-fraud.org was established in July 2012 by the pseudonymous Frances de Triusce (an anagram of “science fraudster”) with the aim of highlighting suspicious papers in the scientific literature. Barely six months later, having brought to light around 500 examples of what might best be called questionable data, and with a daily readership in the thousands, the website was shut down.
Its founder’s true identity had been uncovered - he was Paul Brookes, an associate professor at the University of Rochester Medical Center in the US - and an email had been sent to around 85 scientists whose data had been questioned on the site, encouraging them to sue him for defamation. The email, which described Science-fraud.org as a “hate site” and, rather ironically, as a menace to “scientific society”, was also copied to Brookes’ superiors at Rochester (including its president), the editors of journals in which he had published and prominent people in his field who might be expected to be involved in peer reviewing his grants and papers.
Brookes’ immense frustration with both this deplorable act and the current state of scientific peer review was clear in his final post on the Science Fraud site: “As I have learned the hard way, anyone who dares to stick their neck on the line and question the data of their peers is ostracised, stone walled and subjected to lawsuits.” He went on to argue that the way forward would be to assemble what he called a “coalition of the willing” - to be known as the Association for Anonymous Post-Publication Peer Review - to effectively police the literature, flagging up questionable data and papers. A Science Fraud 2.0, in other words, albeit with a less incendiary name.
I have an immense amount of respect for Brookes, and for all those who will join his coalition. His integrity and commitment to science is laudable and inspiring. But why should he have to stick his neck on the line again? Do we really have to rely on what amounts to academic vigilantism to preserve the integrity of the scientific record? And who decides what constitutes a breach of scientific integrity in any case?
This latter question is, perhaps surprisingly, rather vexed. Even in cases of straightforward fraud - the manipulation, modification, and/or direct fabrication of data - establishing beyond doubt the guilt of the authors is rarely straightforward. But at least journal editors’ responsibility in these circumstances is clear: the paper must be retracted.
A study by medical communications consultant R. Grant Steen of papers retracted from the PubMed database between 2000 and 2010, which was published in the Journal of Medical Ethics in 2010, found that retractions had increased from fewer than 10 in 2000 to close to 180 in 2010. Even more worryingly, in nearly a third of cases journals did not even highlight (via, for example, a watermark on the article) that the paper had been retracted. However, only 27 per cent of the retractions were because of fraud; the rest were attributable either to “undisclosed” reasons or what appears to be genuine scientific error.
But exactly what constitutes scientific error? Should a paper be retracted when there are clear flaws in its methodology? Or when its interpretations are unsupported by the data? Or even when subsequent experiments show its conclusions are incorrect - even if the experimental and theoretical work were carried out to a very high standard? The last suggestion was supported in a widely publicised and very controversial blog posting last year by PLOS Medicine chief editor Virginia Barbour and PLOS Pathogens editor-in-chief Kasturi Haldar. They wrote: “We work with authors…to publish corrections if we find parts of articles to be inaccurate. If a paper’s major conclusions are shown to be wrong we will retract the paper. By doing so, and by being open about our motives, we hope to clarify once and for all that there is no shame in correcting the literature. Despite the best of efforts, errors occur and their timely and effective remedy should be considered the mark of responsible authors, editors and publishers.”
There is much that is laudable in this statement. The journal PLOS One, in particular, has an admirable track record of providing a forum for post- publication critiquing of its articles, and it should also be noted that Barbour is chair of the Committee on Publication Ethics, whose guidelines should be - but currently aren’t - embedded in the codes of practice of all scientific publishers. Yet the suggestion that a paper should be retracted if its conclusions are wrong is a step much too far, and would be baulked at by the majority of scientists.
If a paper’s data are reliable, its methodologies sound and its conclusions plausible at the time and based on the data (rather than authors’ wishful thinking), it is broadly valid and should remain part of the scientific record.
But what if the data and methods aren’t reliable? What if, for example, the researchers are unaware of experimental artefacts that provide a more plausible explanation of their data than the more novel and exciting interpretation they have advanced? Some might argue that the primary responsibility for identifying this type of problem lies with peer reviewers, but, as Richard Feynman said in his 1974 commencement address at the California Institute of Technology, scientists need to go the extra mile in self-criticism before they submit their work for publication.
“If you’re doing an experiment, you should report everything that you think might make it invalid - not only what you think is right about it…The first principle is that you must not fool yourself - and you are the easiest person to fool…After you’ve not fooled yourself, it’s easy not to fool [others],” he said.
It should be said that Feynman’s advice is not entirely consistent with a culture where scientists’ primary goal can too often be to ensure that the paper “gets past the referees”. Nevertheless, it remains received wisdom that science proceeds via a process of self-correction, such that errors will, eventually, be exposed. As comforting a picture as this is, the evidence is stacking up against it.
...
Read more here/Leia mais aqui: The Times Higher Education
