Annals of Emergency Medicine
Longitudinal Trends in the Performance of Scientific Peer Reviewers
Presented at the Sixth International Congress on Peer Review and Biomedical Education, September 2009, Vancouver, British Columbia, Canada.
Michael Callaham, MD, Charles McCulloch, PhD
Received 1 May 2010; received in revised form 9 July 2010; accepted 22 July 2010. published online 15 November 2010.
Corrected Proof
Abstract
Study objective:
We characterize changes in review quality by individual peer reviewers over time.
Methods:
Editors at a specialty journal in the top 11% of Institute of Scientific Information journals rated the quality of every review, using a validated 5-point quality score. Linear mixed-effect models were used to analyze rating changes over time, calculating within-reviewer trends plus predicted slope of change in score for each reviewer. Reviewers at this journal have been shown comparable to those at other journals.
Results:
Reviews (14,808) were performed by 1,499 reviewers and rated by 84 editors during the 14-year study. Ninety-two percent of reviewers demonstrated very slow but steady deterioration in their scores (mean –0.04 points [–0.8%] per year). Rate of deterioration was unrelated to duration of reviewing but moderately correlated with mean reviewer quality score (R=0.52). The mean score of each reviewer's first 4 reviews predicted subsequent performance with a sensitivity of 75% and specificity of 47%. Scores of the group stayed constant over time despite deterioration because newly recruited reviewers initially had higher mean quality scores than their predecessors.
Conclusion:
This study, one of few tracking expert performance longitudinally, demonstrates that most journal peer reviewers received lower quality scores for article assessment over the years. This could be due to deteriorating performance (caused by either cognitive changes or competing priorities) or, to a partial degree, escalating expectations; other explanations were ruled out. This makes monitoring reviewer quality even more crucial to maintain the mission of scientific journals.
+++++
