An Empirical Comparison of the Harmful Effects for Randomized Controlled Trials and Non-randomized Studies of Interventions

Overview

Journal Front Pharmacol

Date 2023 Apr 7

PMID 37025494

Authors

Minhan Dai

Luis Furuya-Kanamori

Asma Syed

Lifeng Lin

Qiang Wang

Affiliations

Soon will be listed here.

Abstract

Randomized controlled trials (RCTs) are the gold standard to evaluate the efficacy of interventions (e.g., drugs and vaccines), yet the sample size of RCTs is often limited for safety assessment. Non-randomized studies of interventions (NRSIs) had been proposed as an important alternative source for safety assessment. In this study, we aimed to investigate whether there is any difference between RCTs and NRSIs in the evaluation of adverse events. We used the dataset of systematic reviews with at least one meta-analysis including both RCTs and NRSIs and collected the 2 × 2 table information (i.e., numbers of cases and sample sizes in intervention and control groups) of each study in the meta-analysis. We matched RCTs and NRSIs by their sample sizes (ratio: 0.85/1 to 1/0.85) within a meta-analysis. We estimated the ratio of the odds ratios (RORs) of an NRSI against an RCT in each pair and used the inverse variance as the weight to combine the natural logarithm of ROR (lnROR). We included systematic reviews with 178 meta analyses, from which we confirmed 119 pairs of RCTs and NRSIs. The pooled ROR of NRSIs compared to that of RCTs was estimated to be 0.96 (95% confidence interval: 0.87 and 1.07). Similar results were obtained with different sample size subgroups and treatment subgroups. With the increase in sample size, the difference in ROR between RCTs and NRSIs decreased, although not significantly. There was no substantial difference in the effects between RCTs and NRSIs in safety assessment when they have similar sample sizes. Evidence from NRSIs might be considered a supplement to RCTs for safety assessment.

References

Ju K, Lin L, Chu H, Cheng L, Xu C . Laplace approximation, penalized quasi-likelihood, and adaptive Gauss-Hermite quadrature for generalized linear mixed models: towards meta-analysis of binary outcome with sparse data. BMC Med Res Methodol. 2020; 20(1):152. PMC: 7296731. DOI: 10.1186/s12874-020-01035-6. View

Yoon B, Ha Y, Lee Y, Koo K . Postoperative Deep Infection After Cemented Versus Cementless Total Hip Arthroplasty: A Meta-Analysis. J Arthroplasty. 2015; 30(10):1823-7. DOI: 10.1016/j.arth.2015.04.041. View

Zhang Y, Ma L . Effect of preoperative angiotensin-converting enzyme inhibitor on the outcome of coronary artery bypass graft surgery. Eur J Cardiothorac Surg. 2014; 47(5):788-95. DOI: 10.1093/ejcts/ezu298. View

Geng H, Nasier D, Liu B, Gao H, Xu Y . Meta-analysis of elective surgical complications related to defunctioning loop ileostomy compared with loop colostomy after low anterior resection for rectal carcinoma. Ann R Coll Surg Engl. 2015; 97(7):494-501. PMC: 5210131. DOI: 10.1308/003588415X14181254789240. View

Col N, Kim J, Chlebowski R . Menopausal hormone therapy after breast cancer: a meta-analysis and critical appraisal of the evidence. Breast Cancer Res. 2005; 7(4):R535-40. PMC: 1175064. DOI: 10.1186/bcr1035. View

Keir A, Pal S, Trivella M, Lieberman L, Callum J, Shehata N . Adverse effects of red blood cell transfusions in neonates: a systematic review and meta-analysis. Transfusion. 2016; 56(11):2773-2780. DOI: 10.1111/trf.13785. View

Doi S, Furuya-Kanamori L, Xu C, Chivese T, Lin L, Musa O . The Odds Ratio is "portable" across baseline risk but not the Relative Risk: Time to do away with the log link in binomial regression. J Clin Epidemiol. 2021; 142:288-293. DOI: 10.1016/j.jclinepi.2021.08.003. View

Touze E, Trinquart L, Chatellier G, Mas J . Systematic review of the perioperative risks of stroke or death after carotid angioplasty and stenting. Stroke. 2009; 40(12):e683-93. DOI: 10.1161/STROKEAHA.109.562041. View

Shah K, Chaker Z, Busu T, Shah R, Osman M, Alqahtani F . Meta-Analysis Comparing Renal Outcomes after Transcatheter versus Surgical Aortic Valve Replacement. J Interv Cardiol. 2019; 2019:3537256. PMC: 6739771. DOI: 10.1155/2019/3537256. View

10.

Golder S, Loke Y, Wright K, Norman G . Reporting of Adverse Events in Published and Unpublished Studies of Health Care Interventions: A Systematic Review. PLoS Med. 2016; 13(9):e1002127. PMC: 5029817. DOI: 10.1371/journal.pmed.1002127. View

11.

McNamee R . Regression modelling and other methods to control confounding. Occup Environ Med. 2005; 62(7):500-6, 472. PMC: 1741049. DOI: 10.1136/oem.2002.001115. View

12.

Jreich R, Sebastien B . Comparison of statistical methodologies used to estimate the treatment effect on time-to-event outcomes in observational studies. J Biopharm Stat. 2021; 31(4):469-489. DOI: 10.1080/10543406.2021.1918140. View

13.

Donner A . Approaches to sample size estimation in the design of clinical trials--a review. Stat Med. 1984; 3(3):199-214. DOI: 10.1002/sim.4780030302. View

14.

Dechartres A, Atal I, Riveros C, Meerpohl J, Ravaud P . Association Between Publication Characteristics and Treatment Effect Estimates: A Meta-epidemiologic Study. Ann Intern Med. 2018; 169(6):385-393. DOI: 10.7326/M18-1517. View

15.

Doi S, Furuya-Kanamori L, Xu C, Lin L, Chivese T, Thalib L . Controversy and Debate: Questionable utility of the relative risk in clinical research: Paper 1: A call for change to practice. J Clin Epidemiol. 2020; 142:271-279. DOI: 10.1016/j.jclinepi.2020.08.019. View

16.

Guyatt G, Oxman A, Vist G, Kunz R, Falck-Ytter Y, Alonso-Coello P . GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008; 336(7650):924-6. PMC: 2335261. DOI: 10.1136/bmj.39489.470347.AD. View

17.

Jia P, Lin L, Kwong J, Xu C . Many meta-analyses of rare events in the Cochrane Database of Systematic Reviews were underpowered. J Clin Epidemiol. 2020; 131:113-122. DOI: 10.1016/j.jclinepi.2020.11.017. View

18.

Yaghoobi M, Farrokhyar F, Yuan Y, Hunt R . Is there an increased risk of GERD after Helicobacter pylori eradication?: a meta-analysis. Am J Gastroenterol. 2010; 105(5):1007-13. DOI: 10.1038/ajg.2009.734. View

19.

Abraham N, Byrne C, Young J, Solomon M . Meta-analysis of well-designed nonrandomized comparative studies of surgical procedures is as good as randomized controlled trials. J Clin Epidemiol. 2009; 63(3):238-45. DOI: 10.1016/j.jclinepi.2009.04.005. View

20.

Hemkens L, Contopoulos-Ioannidis D, Ioannidis J . Agreement of treatment effects for mortality from routinely collected data and subsequent randomized trials: meta-epidemiological survey. BMJ. 2016; 352:i493. PMC: 4772787. DOI: 10.1136/bmj.i493. View