Implementation of a Journal Peer Reviewer Stratification System Based on Quality and Reliability

Study Objective: Before starting this study, Annals of Emergency Medicine had a large and unwieldy reviewer pool that demonstrated substantial variability in quality and reliability. We hypothesize that a tiered reviewer stratification system might enable our journal editors to target the bulk of their review invitations to our better reviewers and thus improve our efficiency.

Methods: In 2003, we instituted a 3-tiered hierarchic classification for our reviewers and stratified them within these categories according to predefined criteria for reviewer quality and reliability. Our approximately 50 editors then targeted the bulk of their review invitations to the top performance tier.

Results: Comparing 2009 data with 2002 (the year before the system), we found fewer late reviews (13% versus 32%) and fewer reviewers not used in a given year (28% versus 59%). More top-tier reviewer invitations led to an on-time review (48% versus 37%) in 2009 compared with 2002. Editors have found the system to be simple and easy to use. No serious problems have been identified.

Conclusion: Implementation of a tiered system stratifying journal peer reviewers by quality and reliability was readily accomplished by Annals and has appeared to improve the efficiency of our peer review.