Reviewing the Review: a Qualitative Assessment of the Peer Review Process in Surgical Journals

Overview

Journal Res Integr Peer Rev

Publisher Biomed Central

Date 2018 Jun 1

PMID 29850109

Citations 8

Authors

Catherine H Davis

Barbara L Bass

Kevin E Behrns

Keith D Lillemoe

O James Garden

Mark S Roh

Jeffrey E Lee

Charles M Balch

Thomas A Aloia

Affiliations

Soon will be listed here.

Abstract

Background: Despite rapid growth of the scientific literature, no consensus guidelines have emerged to define the optimal criteria for editors to grade submitted manuscripts. The purpose of this project was to assess the peer reviewer metrics currently used in the surgical literature to evaluate original manuscript submissions.

Methods: Manuscript grading forms for 14 of the highest circulation general surgery-related journals were evaluated for content, including the type and number of quantitative and qualitative questions asked of peer reviewers. Reviewer grading forms for the seven surgical journals with the higher impact factors were compared to the seven surgical journals with lower impact factors using Fisher's exact tests.

Results: Impact factors of the studied journals ranged from 1.73 to 8.57, with a median impact factor of 4.26 in the higher group and 2.81 in the lower group. The content of the grading forms was found to vary considerably. Relatively few journals asked reviewers to grade specific components of a manuscript. Higher impact factor journal manuscript grading forms more frequently addressed statistical analysis, ethical considerations, and conflict of interest. In contrast, lower impact factor journals more commonly requested reviewers to make qualitative assessments of novelty/originality, scientific validity, and scientific importance.

Conclusion: Substantial variation exists in the grading criteria used to evaluate original manuscripts submitted to the surgical literature for peer review, with differential emphasis placed on certain criteria correlated to journal impact factors.

Citing Articles

Valuing contributions to peer review: A shared responsibility.

Murphy C, Thomas F J Spinal Cord Med. 2025; 48(1):1-2.

PMID: 39819584 PMC: 11749009. DOI: 10.1080/10790268.2025.2441085.

A health sciences researcher's experience of manuscript review comments, 2020-2022.

Joubert G S Afr Fam Pract (2004). 2023; 65(1):e1-e5.

PMID: 37916700 PMC: 10623586. DOI: 10.4102/safp.v65i1.5753.

How to Review a Surgical Scientific Paper: A Guide for Critical Appraisal.

Greco S, Davis C, Hicks C, Kaye A, Maxwell J, Salles A Ann Surg Open. 2023; 2(1):e027.

PMID: 37638253 PMC: 10455126. DOI: 10.1097/AS9.0000000000000027.

Conflict of interest in the peer review process: A survey of peer review reports.

Makarem A, Mroue R, Makarem H, Diab L, Hassan B, Khabsa J PLoS One. 2023; 18(6):e0286908.

PMID: 37289790 PMC: 10249818. DOI: 10.1371/journal.pone.0286908.

A scoping review on biomedical journal peer review guides for reviewers.

Song E, Ang L, Park J, Jun E, Kim K, Jun J PLoS One. 2021; 16(5):e0251440.

PMID: 34014958 PMC: 8136639. DOI: 10.1371/journal.pone.0251440.

References

Manchikanti L, Kaye A, Boswell M, Hirsch J . Medical journal peer review: process and bias. Pain Physician. 2015; 18(1):E1-E14. View

Chung K, Shauver M, Malay S, Zhong L, Weinstein A, Rohrich R . Is Double-Blinded Peer Review Necessary? The Effect of Blinding on Review Quality. Plast Reconstr Surg. 2015; 136(6):1369-1377. DOI: 10.1097/PRS.0000000000001820. View

Seals D, Tanaka H . Manuscript peer review: a helpful checklist for students and novice referees. Adv Physiol Educ. 2000; 23(1):52-8. DOI: 10.1152/advances.2000.23.1.S52. View

von Elm E, Altman D, Egger M, Pocock S, Gotzsche P, Vandenbroucke J . The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Lancet. 2007; 370(9596):1453-7. DOI: 10.1016/S0140-6736(07)61602-X. View

Stahel P, Moore E . How to review a surgical paper: a guide for junior referees. BMC Med. 2016; 14:29. PMC: 4753042. DOI: 10.1186/s12916-016-0578-6. View

Patel J . Why training and specialization is needed for peer review: a case study of peer review for randomized controlled trials. BMC Med. 2014; 12:128. PMC: 4243268. DOI: 10.1186/s12916-014-0128-z. View

Liberati A, Altman D, Tetzlaff J, Mulrow C, Gotzsche P, Ioannidis J . The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol. 2009; 62(10):e1-34. DOI: 10.1016/j.jclinepi.2009.06.006. View

McCulloch P, Feinberg J, Philippou Y, Kolias A, Kehoe S, Lancaster G . Progress in clinical research in surgery and IDEAL. Lancet. 2018; 392(10141):88-94. DOI: 10.1016/S0140-6736(18)30102-8. View

Van Rooyen S, Godlee F, Evans S, Smith R, Black N . Effect of blinding and unmasking on the quality of peer review: a randomized trial. JAMA. 1998; 280(3):234-7. DOI: 10.1001/jama.280.3.234. View

10.

Weller A . Editorial peer review in US medical journals. JAMA. 1990; 263(10):1344-7. View

11.

Godlee F, Gale C, Martyn C . Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. JAMA. 1998; 280(3):237-40. DOI: 10.1001/jama.280.3.237. View

12.

Smith R . Peer review: a flawed process at the heart of science and journals. J R Soc Med. 2006; 99(4):178-82. PMC: 1420798. DOI: 10.1177/014107680609900414. View

13.

Benos D, Bashari E, Chaves J, Gaggar A, Kapoor N, Lafrance M . The ups and downs of peer review. Adv Physiol Educ. 2007; 31(2):145-52. DOI: 10.1152/advan.00104.2006. View

14.

Burnham J . The evolution of editorial peer review. JAMA. 1990; 263(10):1323-9. View

15.

Frank E . Editors' requests of peer reviewers: a study and a proposal. Prev Med. 1996; 25(2):102-4. DOI: 10.1006/pmed.1996.0035. View

16.

McFadden D, LeMaire S, Sarr M, Behrns K . How to review a Paper: Suggestions from the Editors of Surgery and the Journal of Surgical Research. Surgery. 2017; 162(1):1-6. DOI: 10.1016/j.surg.2017.05.001. View

17.

Stahel P, Moore E . Peer review for biomedical publications: we can improve the system. BMC Med. 2014; 12:179. PMC: 4177268. DOI: 10.1186/s12916-014-0179-1. View

18.

Agha R, Barai I, Rajmohan S, Lee S, Anwar M, Fowler A . Support for reporting guidelines in surgical journals needs improvement: A systematic review. Int J Surg. 2017; 45:14-17. DOI: 10.1016/j.ijsu.2017.06.084. View

19.

Moher D, Hopewell S, Schulz K, Montori V, Gotzsche P, Devereaux P . CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ. 2010; 340:c869. PMC: 2844943. DOI: 10.1136/bmj.c869. View

20.

McNutt R, Evans A, Fletcher R, Fletcher S . The effects of blinding on the quality of peer review. A randomized trial. JAMA. 1990; 263(10):1371-6. View