Does Mentoring New Peer Reviewers Improve Review Quality? A Randomized Trial

Overview

Journal BMC Med Educ

Publisher Biomed Central

Specialty Medical Education

Date 2012 Aug 30

PMID 22928960

Citations 23

Authors

Debra Houry

Steven Green

Michael Callaham

Affiliations

Soon will be listed here.

Abstract

Background: Prior efforts to train medical journal peer reviewers have not improved subsequent review quality, although such interventions were general and brief. We hypothesized that a manuscript-specific and more extended intervention pairing new reviewers with high-quality senior reviewers as mentors would improve subsequent review quality.

Methods: Over a four-year period we randomly assigned all new reviewers for Annals of Emergency Medicine to receive our standard written informational materials alone, or these materials plus a new mentoring intervention. For this program we paired new reviewers with a high-quality senior reviewer for each of their first three manuscript reviews, and asked mentees to discuss their review with their mentor by email or phone. We then compared the quality of subsequent reviews between the control and intervention groups, using linear mixed effects models of the slopes of review quality scores over time.

Results: We studied 490 manuscript reviews, with similar baseline characteristics between the 24 mentees who completed the trial and the 22 control reviewers. Mean quality scores for the first 3 reviews on our 1 to 5 point scale were similar between control and mentee groups (3.4 versus 3.5), as were slopes of change of review scores (-0.229 versus -0.549) and all other secondary measures of reviewer performance.

Conclusions: A structured training intervention of pairing newly recruited medical journal peer reviewers with senior reviewer mentors did not improve the quality of their subsequent reviews.

Citing Articles

A structured, journal-led peer-review mentoring program enhances peer review training.

Lyons-Warren A, Aamodt W, Pieper K, Strowd R Res Integr Peer Rev. 2024; 9(1):3.

PMID: 38454514 PMC: 10921741. DOI: 10.1186/s41073-024-00143-x.

The inaugural Junior Editorial Board-lessons learned and the path forward toward improving the peer review process.

Adinortey C, Dolan S, Doore S, Lijek R, Pires D, Yu W mBio. 2023; 15(1):e0199123.

PMID: 38099716 PMC: 10790782. DOI: 10.1128/mbio.01991-23.

Reviewer training for improving grant and journal peer review.

Hesselberg J, Dalsbo T, Stromme H, Svege I, Fretheim A Cochrane Database Syst Rev. 2023; 11:MR000056.

PMID: 38014743 PMC: 10683016. DOI: 10.1002/14651858.MR000056.pub2.

Preprint Peer Review Enhances Undergraduate Biology Students' Disciplinary Literacy and Sense of Belonging in STEM.

Otto J, McDowell G, Balgopal M, Lijek R J Microbiol Biol Educ. 2023; 24(2).

PMID: 37614887 PMC: 10443316. DOI: 10.1128/jmbe.00053-23.

Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review.

Hosseini M, Horbach S Res Integr Peer Rev. 2023; 8(1):4.

PMID: 37198671 PMC: 10191680. DOI: 10.1186/s41073-023-00133-5.

References

Evans A, McNutt R, Fletcher S, Fletcher R . The characteristics of peer reviewers who produce good-quality reviews. J Gen Intern Med. 1993; 8(8):422-8. DOI: 10.1007/BF02599618. View

Shashok K . Content and communication: how can peer review provide helpful feedback about the writing?. BMC Med Res Methodol. 2008; 8:3. PMC: 2268697. DOI: 10.1186/1471-2288-8-3. View

Callaham M, Schriger D . Effect of structured workshop training on subsequent performance of journal peer reviewers. Ann Emerg Med. 2002; 40(3):323-8. DOI: 10.1067/mem.2002.127121. View

Van Rooyen S, Black N, Godlee F . Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts. J Clin Epidemiol. 1999; 52(7):625-9. DOI: 10.1016/s0895-4356(99)00047-5. View

Sullivan L, Dukes K, Losina E . Tutorial in biostatistics. An introduction to hierarchical linear modelling. Stat Med. 1999; 18(7):855-88. DOI: 10.1002/(sici)1097-0258(19990415)18:7<855::aid-sim117>3.0.co;2-7. View

Green S, Callaham M . Implementation of a journal peer reviewer stratification system based on quality and reliability. Ann Emerg Med. 2010; 57(2):149-152.e4. DOI: 10.1016/j.annemergmed.2010.08.005. View

Schroter S, Black N, Evans S, Carpenter J, Godlee F, Smith R . Effects of training on quality of peer review: randomised controlled trial. BMJ. 2004; 328(7441):673. PMC: 381220. DOI: 10.1136/bmj.38023.700775.AE. View

Callaham M, Wears R, Waeckerle J . Effect of attendance at a training session on peer reviewer quality and performance. Ann Emerg Med. 1998; 32(3 Pt 1):318-22. DOI: 10.1016/s0196-0644(98)70007-1. View

Taylor R, Reeves B, Ewings P, Taylor R . Critical appraisal skills training for health care professionals: a randomized controlled trial [ISRCTN46272378]. BMC Med Educ. 2004; 4(1):30. PMC: 539272. DOI: 10.1186/1472-6920-4-30. View

10.

Callaham M, Baxt W, Waeckerle J, Wears R . Reliability of editors' subjective quality ratings of peer reviews of manuscripts. JAMA. 1998; 280(3):229-31. DOI: 10.1001/jama.280.3.229. View

11.

Freda M, Kearney M, Baggs J, Broome M, Dougherty M . Peer reviewer training and editor support: results from an international survey of nursing peer reviewers. J Prof Nurs. 2009; 25(2):101-8. DOI: 10.1016/j.profnurs.2008.08.007. View

12.

Callaham M, Tercier J . The relationship of previous training and experience of journal peer reviewers to subsequent review quality. PLoS Med. 2007; 4(1):e40. PMC: 1796627. DOI: 10.1371/journal.pmed.0040040. View

13.

Callaham M, Knopp R, Gallagher E . Effect of written feedback by editors on quality of reviews: two randomized trials. JAMA. 2002; 287(21):2781-3. DOI: 10.1001/jama.287.21.2781. View

14.

Callaham M, McCulloch C . Longitudinal trends in the performance of scientific peer reviewers. Ann Emerg Med. 2010; 57(2):141-8. DOI: 10.1016/j.annemergmed.2010.07.027. View

15.

Ebbert J, Montori V, SCHULTZ H . The journal club in postgraduate medical education: a systematic review. Med Teach. 2002; 23(5):455-61. DOI: 10.1080/01421590120075670. View

16.

Parkes J, Hyde C, Deeks J, Milne R . Teaching critical appraisal skills in health care settings. Cochrane Database Syst Rev. 2001; (3):CD001270. DOI: 10.1002/14651858.CD001270. View

17.

Snell L, Spencer J . Reviewers' perceptions of the peer review process for a medical education journal. Med Educ. 2004; 39(1):90-7. DOI: 10.1111/j.1365-2929.2004.02026.x. View