GRADE Guidelines: 18. How ROBINS-I and Other Tools to Assess Risk of Bias in Nonrandomized Studies Should Be Used to Rate the Certainty of a Body of Evidence

Overview

Journal J Clin Epidemiol

Publisher Elsevier

Specialty Public Health

Date 2018 Feb 13

PMID 29432858

Citations 283

Authors

Holger J Schunemann

Carlos Cuello

Elie A Akl

Reem A Mustafa

Jorg J Meerpohl

Kris Thayer

Rebecca L Morgan

Gerald Gartlehner

Regina Kunz

S Vittal Katikireddi

Jonathan Sterne

Julian Pt Higgins

Gordon Guyatt

Affiliations

Soon will be listed here.

Abstract

Objective: To provide guidance on how systematic review authors, guideline developers, and health technology assessment practitioners should approach the use of the risk of bias in nonrandomized studies of interventions (ROBINS-I) tool as a part of GRADE's certainty rating process.

Study Design And Setting: The study design and setting comprised iterative discussions, testing in systematic reviews, and presentation at GRADE working group meetings with feedback from the GRADE working group.

Results: We describe where to start the initial assessment of a body of evidence with the use of ROBINS-I and where one would anticipate the final rating would end up. The GRADE accounted for issues that mitigate concerns about confounding and selection bias by introducing the upgrading domains: large effects, dose-effect relations, and when plausible residual confounders or other biases increase certainty. They will need to be considered in an assessment of a body of evidence when using ROBINS-I.

Conclusions: The use of ROBINS-I in GRADE assessments may allow for a better comparison of evidence from randomized controlled trials (RCTs) and nonrandomized studies (NRSs) because they are placed on a common metric for risk of bias. Challenges remain, including appropriate presentation of evidence from RCTs and NRSs for decision-making and how to optimally integrate RCTs and NRSs in an evidence assessment.

Citing Articles

Rituximab for people with multiple sclerosis.

Filippini G, Kruja J, Del Giovane C Cochrane Database Syst Rev. 2025; 3:CD013874.

PMID: 40066932 PMC: 11895426. DOI: 10.1002/14651858.CD013874.pub3.

Nurse-Led Electrical External Cardioversion of Patients with Atrial Arrhythmia: A Systematic Review Update and Meta-Analysis.

Caleffi D, Pingani L, Rovesti S, Cannizzaro D, Ferri P Nurs Rep. 2025; 15(2).

PMID: 39997768 PMC: 11858258. DOI: 10.3390/nursrep15020032.

Relative Efficacy, Effectiveness and Safety of Newer and/or Enhanced Seasonal Influenza Vaccines for the Prevention of Laboratory-Confirmed Influenza in Individuals Aged 18 years and Over: Update of a Systematic Review.

Askar M, Ali K, Batke M, Brugger T, Falman A, Robertson A Rev Med Virol. 2025; 35(2):e70020.

PMID: 39994168 PMC: 11850296. DOI: 10.1002/rmv.70020.

Decoding the brain-machine interaction for upper limb assistive technologies: advances and challenges.

Ghosh S, Yadav R, Soni S, Giri S, Muthukrishnan S, Kumar L Front Hum Neurosci. 2025; 19:1532783.

PMID: 39981127 PMC: 11839673. DOI: 10.3389/fnhum.2025.1532783.

Early versus late tracheostomy in people with multiple trauma.

Ansems K, Aleksandrova E, Steinfeld E, Metzendorf M, Skoetz N, Benstoem C Cochrane Database Syst Rev. 2025; 5:CD015932.

PMID: 39908070 PMC: 11091947. DOI: 10.1002/14651858.CD015932.

References

Guyatt G, Oxman A, Sultan S, Brozek J, Glasziou P, Alonso-Coello P . GRADE guidelines: 11. Making an overall rating of confidence in effect estimates for a single outcome and for all outcomes. J Clin Epidemiol. 2012; 66(2):151-7. DOI: 10.1016/j.jclinepi.2012.01.006. View

Craig P, Katikireddi S, Leyland A, Popham F . Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research. Annu Rev Public Health. 2017; 38:39-56. PMC: 6485604. DOI: 10.1146/annurev-publhealth-031816-044327. View

Schunemann H, Best D, Vist G, Oxman A . Letters, numbers, symbols and words: how to communicate grades of evidence and recommendations. CMAJ. 2003; 169(7):677-80. PMC: 202287. View

Katikireddi S, Egan M, Petticrew M . How do systematic reviews incorporate risk of bias assessments into the synthesis of evidence? A methodological study. J Epidemiol Community Health. 2014; 69(2):189-95. PMC: 4316857. DOI: 10.1136/jech-2014-204711. View

Schunemann H, Oxman A, Brozek J, Glasziou P, Bossuyt P, Chang S . GRADE: assessing the quality of evidence for diagnostic recommendations. ACP J Club. 2008; 149(6):2. View

Guyatt G, Oxman A, Sultan S, Glasziou P, Akl E, Alonso-Coello P . GRADE guidelines: 9. Rating up the quality of evidence. J Clin Epidemiol. 2011; 64(12):1311-6. DOI: 10.1016/j.jclinepi.2011.06.004. View

Sterne J, Hernan M, Reeves B, Savovic J, Berkman N, Viswanathan M . ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016; 355:i4919. PMC: 5062054. DOI: 10.1136/bmj.i4919. View

Devereaux P, Choi P, Lacchetti C, Weaver B, Schunemann H, Haines T . A systematic review and meta-analysis of studies comparing mortality rates of private for-profit and private not-for-profit hospitals. CMAJ. 2002; 166(11):1399-406. PMC: 111211. View

Guyatt G, Oxman A, Akl E, Kunz R, Vist G, Brozek J . GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011; 64(4):383-94. DOI: 10.1016/j.jclinepi.2010.04.026. View

10.

Hultcrantz M, Rind D, Akl E, Treweek S, Mustafa R, Iorio A . The GRADE Working Group clarifies the construct of certainty of evidence. J Clin Epidemiol. 2017; 87:4-13. PMC: 6542664. DOI: 10.1016/j.jclinepi.2017.05.006. View

11.

Alonso-Coello P, Schunemann H, Moberg J, Brignardello-Petersen R, Akl E, Davoli M . GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ. 2016; 353:i2016. DOI: 10.1136/bmj.i2016. View

12.

Guyatt G, Oxman A, Schunemann H, Tugwell P, Knottnerus A . GRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology. J Clin Epidemiol. 2010; 64(4):380-2. DOI: 10.1016/j.jclinepi.2010.09.011. View

13.

Sanderson S, Tatt I, Higgins J . Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidemiol. 2007; 36(3):666-76. DOI: 10.1093/ije/dym018. View

14.

Schunemann H, Hill S, Kakad M, Vist G, Bellamy R, Stockman L . Transparent development of the WHO rapid advice guidelines. PLoS Med. 2007; 4(5):e119. PMC: 1877972. DOI: 10.1371/journal.pmed.0040119. View

15.

Morgan R, Thayer K, Bero L, Bruce N, Falck-Ytter Y, Ghersi D . GRADE: Assessing the quality of evidence in environmental and occupational health. Environ Int. 2016; 92-93:611-6. PMC: 4902742. DOI: 10.1016/j.envint.2016.01.004. View

16.

Schunemann H, Schunemann A, Oxman A, Brozek J, Glasziou P, Jaeschke R . Grading quality of evidence and strength of recommendations for diagnostic tests and strategies. BMJ. 2008; 336(7653):1106-10. PMC: 2386626. DOI: 10.1136/bmj.39500.677199.AE. View

17.

Iorio A, Spencer F, Falavigna M, Alba C, Lang E, Burnand B . Use of GRADE for assessment of evidence about prognosis: rating confidence in estimates of event rates in broad categories of patients. BMJ. 2015; 350:h870. DOI: 10.1136/bmj.h870. View

18.

Burford B, Rehfuess E, Schunemann H, Akl E, Waters E, Armstrong R . Assessing evidence in public health: the added value of GRADE. J Public Health (Oxf). 2012; 34(4):631-5. DOI: 10.1093/pubmed/fds092. View

19.

Spencer F, Iorio A, You J, Murad M, Schunemann H, Vandvik P . Uncertainties in baseline risk estimates and confidence in treatment effects. BMJ. 2012; 345:e7401. DOI: 10.1136/bmj.e7401. View

20.

Atkins D, Best D, Briss P, Eccles M, Falck-Ytter Y, Flottorp S . Grading quality of evidence and strength of recommendations. BMJ. 2004; 328(7454):1490. PMC: 428525. DOI: 10.1136/bmj.328.7454.1490. View