» Articles » PMID: 30552251

Influence of External Peer Reviewer Scores for Funding Applications on Funding Board Decisions: a Retrospective Analysis of 1561 Reviews

Overview
Journal BMJ Open
Specialty General Medicine
Date 2018 Dec 16
PMID 30552251
Citations 5
Authors
Affiliations
Soon will be listed here.
Abstract

Objectives: To evaluate the influence of external peer reviewer scores on the National Institute for Health Research (NIHR) research funding board decisions by the number of reviewers and type of reviewer expertise.

Design: Retrospective analysis of external peer review scores for shortlisted full applications for funding (280 funding applications, 1236 individual reviewers, 1561 review scores).

Setting: Four applied health research funding programmes of NIHR, UK.

Main Outcome Measures: Board decision to fund or not fund research applications.

Results: The mean score of reviewers predicted funding decisions better than individual reviewer scores (area under the receiver operating characteristic (ROC) curve 0.75, 95% CI 0.69 to 0.81 compared with 0.62, CI 0.59 to 0.65). There was no substantial improvement in how accurately mean reviewer scores predicted funding decisions when the number of reviewers increased above 4 (area under ROC curve 0.75, CI 0.59 to 0.91 for four reviewers; 0.80, CI 0.67 to 0.92 for seven or more). Reviewers with differing expertise influenced the board's decision equally, including public and patient reviewers (area under ROC curves from 0.57, CI 0.47 to 0.66 for health economists to 0.64, CI 0.57 to 0.70 for subject-matter experts). The areas under the ROC curves were quite low when using reviewers' scores, confirming that boards do not rely solely on those scores alone to make their funding decisions, which are best predicted by the mean board score.

Conclusions: Boards value scores that originate from a diverse pool of reviewers. On the basis of independent reviewer score alone, there is no detectable benefit of using more than four reviewer scores in terms of their influence on board decisions, so to improve efficiency, it may be possible to avoid using larger numbers of reviewers. The funding decision is best predicted by the board score.

Citing Articles

Getting funded in a highly fluctuating environment: Shifting from excellence to luck and timing.

Kindsiko E, Roigas K, Niinemets U PLoS One. 2022; 17(11):e0277337.

PMID: 36342950 PMC: 9639839. DOI: 10.1371/journal.pone.0277337.


Decision-making approaches used by UK and international health funding organisations for allocating research funds: A survey of current practice.

Meadmore K, Fackrell K, Recio-Saucedo A, Bull A, Fraser S, Blatch-Jones A PLoS One. 2020; 15(11):e0239757.

PMID: 33151954 PMC: 7644005. DOI: 10.1371/journal.pone.0239757.


Assessing health research grant applications: A retrospective comparative review of a one-stage versus a two-stage application assessment process.

Morgan B, Yu L, Solomon T, Ziebland S PLoS One. 2020; 15(3):e0230118.

PMID: 32163468 PMC: 7067561. DOI: 10.1371/journal.pone.0230118.


The CTSA External Reviewer Exchange Consortium (CEREC): Engagement and efficacy.

Schneider M, Bagaporo A, Croker J, Davidson A, Dillon P, Dinkjian A J Clin Transl Sci. 2019; 3(6):325-331.

PMID: 31827906 PMC: 6885993. DOI: 10.1017/cts.2019.411.


Evaluation of stakeholder views on peer review of NIHR applications for funding: a qualitative study.

Turner S, Bull A, Chinnery F, Hinks J, Mcardle N, Moran R BMJ Open. 2018; 8(12):e022548.

PMID: 30552252 PMC: 6303555. DOI: 10.1136/bmjopen-2018-022548.

References
1.
Bland J, Altman D . Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986; 1(8476):307-10. View

2.
Sattler D, McKnight P, Naney L, Mathis R . Grant Peer Review: Improving Inter-Rater Reliability with Training. PLoS One. 2015; 10(6):e0130450. PMC: 4468126. DOI: 10.1371/journal.pone.0130450. View

3.
Mayo N, Brophy J, Goldberg M, Klein M, Miller S, Platt R . Peering at peer review revealed high degree of chance associated with funding of grant applications. J Clin Epidemiol. 2006; 59(8):842-8. DOI: 10.1016/j.jclinepi.2005.12.007. View

4.
Bland J, Altman D . Measuring agreement in method comparison studies. Stat Methods Med Res. 1999; 8(2):135-60. DOI: 10.1177/096228029900800204. View

5.
Bonetta L . Enhancing NIH grant peer review: a broader perspective. Cell. 2008; 135(2):201-4. DOI: 10.1016/j.cell.2008.09.051. View