A Risk of Bias Instrument for Non-randomized Studies of Exposures: A Users' Guide to Its Application in the Context of GRADE

Overview

Journal Environ Int

Specialties Environmental Health
Toxicology

Date 2018 Nov 27

PMID 30473382

Citations 95

Authors

Rebecca L Morgan

Kristina A Thayer

Nancy Santesso

Alison C Holloway

Robyn Blain

Sorina E Eftim

Alexandra E Goldstone

Pam Ross

Mohammed Ansari

Elie A Akl

Tommaso Filippini

Anna Hansell

Joerg J Meerpohl

Reem A Mustafa

Jos Verbeek

Marco Vinceti

Paul Whaley

Holger J Schunemann

Affiliations

Soon will be listed here.

Abstract

The objective of this paper is to explain how to apply, interpret, and present the results of a new instrument to assess the risk of bias (RoB) in non-randomized studies (NRS) dealing with effects of environmental exposures on health outcomes. This instrument is modeled on the Risk Of Bias In Non-randomized Studies of Interventions (ROBINS-I) instrument. The RoB instrument for NRS of exposures assesses RoB along a standardized comparison to a randomized target experiment, instead of the study-design directed RoB approach. We provide specific guidance for the integral steps of developing a research question and target experiment, distinguishing issues of indirectness from RoB, making individual-study judgments, and performing and interpreting sensitivity analyses for RoB judgments across a body of evidence. Also, we present an approach for integrating the RoB assessments within the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) framework to assess the certainty of the evidence in the systematic review. Finally, we guide the reader through an overall assessment to support the rating of all domains that determine the certainty of a body of evidence using the GRADE approach.

Citing Articles

Factors associated with prehospital and in-hospital delays in acute ischaemic stroke care in Indonesia: a systematic review.

Wiyarta E, Hidayat R, Kurniawan M, Sinaga G, Budiman R BMJ Open. 2025; 15(3):e095845.

PMID: 40074251 PMC: 11904354. DOI: 10.1136/bmjopen-2024-095845.

Real-World Data and Real-World Evidence in Regulatory Decision Making: Report Summary From the Council for International Organizations of Medical Sciences (CIOMS) Working Group XIII.

Hennessy S, Atsuta Y, Hill S, Rago L, Juhaeri J Pharmacoepidemiol Drug Saf. 2025; 34(3):e70117.

PMID: 40070104 PMC: 11897686. DOI: 10.1002/pds.70117.

Night shift work and breast cancer risk - 2023 update of epidemiologic evidence.

Hansen J, Pedersen J J Natl Cancer Cent. 2025; 5(1):94-103.

PMID: 40040870 PMC: 11873625. DOI: 10.1016/j.jncc.2024.07.004.

Association of diethylhexyl phthalate exposure with serum thyroid hormone levels: a systematic review and meta-analysis.

Xu K, Sun H, Li J, He X, Zhou X, Cheng X Am J Transl Res. 2025; 16(12):7884-7897.

PMID: 39822510 PMC: 11733379. DOI: 10.62347/IUMS4514.

Enhancing recovery and reducing inflammation: the impact of enhanced recovery after surgery recommendations on inflammatory markers in laparoscopic surgery-a scoping review.

Alves Bersot C, Ferreira Gomes Pereira L, Goncho V, Pereira J, Dos Reis Falcao L Front Surg. 2024; 11:1450434.

PMID: 39717352 PMC: 11663872. DOI: 10.3389/fsurg.2024.1450434.

References

Thayer K, Doerge D, Hunt D, Schurman S, Twaddle N, Churchwell M . Pharmacokinetics of bisphenol A in humans following a single oral administration. Environ Int. 2015; 83:107-15. PMC: 4545316. DOI: 10.1016/j.envint.2015.06.008. View

Liberati A, Altman D, Tetzlaff J, Mulrow C, Gotzsche P, Ioannidis J . The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009; 6(7):e1000100. PMC: 2707010. DOI: 10.1371/journal.pmed.1000100. View

Rooney A, Cooper G, Jahnke G, Lam J, Morgan R, Boyles A . How credible are the study results? Evaluating and applying internal validity tools to literature-based assessments of environmental health hazards. Environ Int. 2016; 92-93:617-29. PMC: 4902751. DOI: 10.1016/j.envint.2016.01.005. View

Kogevinas M . Epidemiological approaches in the investigation of environmental causes of cancer: the case of dioxins and water disinfection by-products. Environ Health. 2011; 10 Suppl 1:S3. PMC: 3073195. DOI: 10.1186/1476-069X-10-S1-S3. View

Sterne J, Hernan M, Reeves B, Savovic J, Berkman N, Viswanathan M . ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016; 355:i4919. PMC: 5062054. DOI: 10.1136/bmj.i4919. View

Carwile J, Michels K . Urinary bisphenol A and obesity: NHANES 2003-2006. Environ Res. 2011; 111(6):825-30. PMC: 3747781. DOI: 10.1016/j.envres.2011.05.014. View

Woodruff T, Sutton P . The Navigation Guide systematic review methodology: a rigorous and transparent method for translating environmental health science into better health outcomes. Environ Health Perspect. 2014; 122(10):1007-14. PMC: 4181919. DOI: 10.1289/ehp.1307175. View

Li D, Miao M, Zhou Z, Wu C, Shi H, Liu X . Urine bisphenol-A level in relation to obesity and overweight in school-age children. PLoS One. 2013; 8(6):e65399. PMC: 3680397. DOI: 10.1371/journal.pone.0065399. View

DOLL R, Hill A . MORTALITY IN RELATION TO SMOKING: TEN YEARS' OBSERVATIONS OF BRITISH DOCTORS. Br Med J. 1964; 1(5395):1399-410. PMC: 1814562. DOI: 10.1136/bmj.1.5395.1399. View

10.

Guyatt G, Oxman A, Kunz R, Woodcock J, Brozek J, Helfand M . GRADE guidelines: 8. Rating the quality of evidence--indirectness. J Clin Epidemiol. 2011; 64(12):1303-10. DOI: 10.1016/j.jclinepi.2011.04.014. View

11.

Ranciere F, Lyons J, Loh V, Botton J, Galloway T, Wang T . Bisphenol A and the risk of cardiometabolic disorders: a systematic review with meta-analysis of the epidemiological evidence. Environ Health. 2015; 14:46. PMC: 4472611. DOI: 10.1186/s12940-015-0036-5. View

12.

Schunemann H, Cuello C, Akl E, Mustafa R, Meerpohl J, Thayer K . GRADE guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence. J Clin Epidemiol. 2018; 111:105-114. PMC: 6692166. DOI: 10.1016/j.jclinepi.2018.01.012. View

13.

Balshem H, Helfand M, Schunemann H, Oxman A, Kunz R, Brozek J . GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol. 2011; 64(4):401-6. DOI: 10.1016/j.jclinepi.2010.07.015. View

14.

Morgan R, Whaley P, Thayer K, Schunemann H . Identifying the PECO: A framework for formulating good questions to explore the association of environmental and other exposures with health outcomes. Environ Int. 2018; 121(Pt 1):1027-1031. PMC: 6908441. DOI: 10.1016/j.envint.2018.07.015. View

15.

Morgan R, Thayer K, Santesso N, Holloway A, Blain R, Eftim S . Evaluation of the risk of bias in non-randomized studies of interventions (ROBINS-I) and the 'target experiment' concept in studies of exposures: Rationale and preliminary instrument development. Environ Int. 2018; 120:382-387. PMC: 9581061. DOI: 10.1016/j.envint.2018.08.018. View

16.

Harley K, Aguilar Schall R, Chevrier J, Tyler K, Aguirre H, Bradman A . Prenatal and postnatal bisphenol A exposure and body mass index in childhood in the CHAMACOS cohort. Environ Health Perspect. 2013; 121(4):514-20. PMC: 3620748. DOI: 10.1289/ehp.1205548. View

17.

Wang H, Zhou Y, Tang C, Wu J, Chen Y, Jiang Q . Association between bisphenol A exposure and body mass index in Chinese school children: a cross-sectional study. Environ Health. 2012; 11:79. PMC: 3549797. DOI: 10.1186/1476-069X-11-79. View

18.

Hernan M, Robins J . Using Big Data to Emulate a Target Trial When a Randomized Trial Is Not Available. Am J Epidemiol. 2016; 183(8):758-64. PMC: 4832051. DOI: 10.1093/aje/kwv254. View

19.

Morgan R, Thayer K, Bero L, Bruce N, Falck-Ytter Y, Ghersi D . GRADE: Assessing the quality of evidence in environmental and occupational health. Environ Int. 2016; 92-93:611-6. PMC: 4902742. DOI: 10.1016/j.envint.2016.01.004. View

20.

Blair A, Stewart P, Lubin J, Forastiere F . Methodological issues regarding confounding and exposure misclassification in epidemiological studies of occupational exposures. Am J Ind Med. 2006; 50(3):199-207. DOI: 10.1002/ajim.20281. View