Meta-analyses of Adverse Effects Data Derived from Randomised Controlled Trials As Compared to Observational Studies: Methodological Overview

Overview

Journal PLoS Med

Specialty General Medicine

Date 2011 May 12

PMID 21559325

Citations 111

Authors

Su Golder

Yoon K Loke

Martin Bland

Affiliations

Soon will be listed here.

Abstract

Background: There is considerable debate as to the relative merits of using randomised controlled trial (RCT) data as opposed to observational data in systematic reviews of adverse effects. This meta-analysis of meta-analyses aimed to assess the level of agreement or disagreement in the estimates of harm derived from meta-analysis of RCTs as compared to meta-analysis of observational studies.

Methods And Findings: Searches were carried out in ten databases in addition to reference checking, contacting experts, citation searches, and hand-searching key journals, conference proceedings, and Web sites. Studies were included where a pooled relative measure of an adverse effect (odds ratio or risk ratio) from RCTs could be directly compared, using the ratio of odds ratios, with the pooled estimate for the same adverse effect arising from observational studies. Nineteen studies, yielding 58 meta-analyses, were identified for inclusion. The pooled ratio of odds ratios of RCTs compared to observational studies was estimated to be 1.03 (95% confidence interval 0.93-1.15). There was less discrepancy with larger studies. The symmetric funnel plot suggests that there is no consistent difference between risk estimates from meta-analysis of RCT data and those from meta-analysis of observational studies. In almost all instances, the estimates of harm from meta-analyses of the different study designs had 95% confidence intervals that overlapped (54/58, 93%). In terms of statistical significance, in nearly two-thirds (37/58, 64%), the results agreed (both studies showing a significant increase or significant decrease or both showing no significant difference). In only one meta-analysis about one adverse effect was there opposing statistical significance.

Conclusions: Empirical evidence from this overview indicates that there is no difference on average in the risk estimate of adverse effects of an intervention derived from meta-analyses of RCTs and meta-analyses of observational studies. This suggests that systematic reviews of adverse effects should not be restricted to specific study types. Please see later in the article for the Editors' Summary.

Citing Articles

The relevance of the real-world evidence in research, clinical, and regulatory decision making.

Costa V, Custodio M, Gefen E, Fregni F Front Public Health. 2025; 13:1512429.

PMID: 40041193 PMC: 11878099. DOI: 10.3389/fpubh.2025.1512429.

Glucagon-Like Peptide 1 Receptor Agonists and Chronic Lower Respiratory Disease Among Type 2 Diabetes Patients: Replication and Reliability Assessment Across a Research Network.

Conover M, Albogami Y, Hardin J, Reich C, Ostropolets A, Ryan P Pharmacoepidemiol Drug Saf. 2025; 34(1):e70087.

PMID: 39805811 PMC: 11730806. DOI: 10.1002/pds.70087.

Integration of non-randomized studies with randomized controlled trials in meta-analyses of clinical studies: a meta-epidemiological study on effect estimation of interventions.

Mei F, Yao M, Wang Y, Huan J, Ma Y, Li G BMC Med. 2024; 22(1):571.

PMID: 39623370 PMC: 11613474. DOI: 10.1186/s12916-024-03778-1.

Integrating randomized controlled trials and non-randomized studies of interventions to assess the effect of rare events: a Bayesian re-analysis of two meta-analyses.

Zhou Y, Yao M, Mei F, Ma Y, Huan J, Zou K BMC Med Res Methodol. 2024; 24(1):219.

PMID: 39333867 PMC: 11430109. DOI: 10.1186/s12874-024-02347-7.

Traditional Chinese exercise in chronic obstructive pulmonary disease: An overview of systematic reviews.

Han L, Wang J, Zhu Y, Lu Y, Liu C, Chen C Medicine (Baltimore). 2024; 103(26):e38700.

PMID: 38941382 PMC: 11466204. DOI: 10.1097/MD.0000000000038700.

References

Hall W, Lucke J . How have the selective serotonin reuptake inhibitor antidepressants affected suicide mortality?. Aust N Z J Psychiatry. 2006; 40(11-12):941-50. DOI: 10.1080/j.1440-1614.2006.01917.x. View

Henry D, McGettigan P . Epidemiology overview of gastrointestinal and renal toxicity of NSAIDs. Int J Clin Pract Suppl. 2003; (135):43-9. View

Petitti D . Coronary heart disease and estrogen replacement therapy. Can compliance bias explain the results of observational studies?. Ann Epidemiol. 1994; 4(2):115-8. DOI: 10.1016/1047-2797(94)90056-6. View

Ofman J, Maclean C, Straus W, Morton S, Berger M, Roth E . A metaanalysis of severe upper gastrointestinal complications of nonsteroidal antiinflammatory drugs. J Rheumatol. 2002; 29(4):804-12. View

Bergendal A, Odlind V, Persson I, Kieler H . Limited knowledge on progestogen-only contraception and risk of venous thromboembolism. Acta Obstet Gynecol Scand. 2009; 88(3):261-6. DOI: 10.1080/00016340902730375. View

Kaufman D, Shapiro S . Epidemiological assessment of drug-induced disease. Lancet. 2000; 356(9238):1339-43. DOI: 10.1016/S0140-6736(00)02826-9. View

Loke Y, Singh S, Furberg C . Long-term use of thiazolidinediones and fractures in type 2 diabetes: a meta-analysis. CMAJ. 2008; 180(1):32-9. PMC: 2612065. DOI: 10.1503/cmaj.080486. View

Singh S, Loke Y, Furberg C . Thiazolidinediones and heart failure: a teleo-analysis. Diabetes Care. 2007; 30(8):2148-53. DOI: 10.2337/dc07-0141. View

Jacob R, Lloyd P . How to evaluate a dental article about harm. J Prosthet Dent. 2000; 84(1):8-16. DOI: 10.1067/mpr.2000.108496. View

10.

Chou R, Fu R, Carson S, Saha S, Helfand M . Methodological shortcomings predicted lower harm estimates in one of two sets of studies of clinical interventions. J Clin Epidemiol. 2006; 60(1):18-28. DOI: 10.1016/j.jclinepi.2006.02.021. View

11.

Tramer M, Moore R, McQuay H . Propofol and bradycardia: causation, frequency and severity. Br J Anaesth. 1997; 78(6):642-51. DOI: 10.1093/bja/78.6.642. View

12.

Ioannidis J, Contopoulos-Ioannidis D . Reporting of safety data from randomised trials. Lancet. 1998; 352(9142):1752-3. DOI: 10.1016/S0140-6736(05)79825-1. View

13.

Edwards J, McQuay H, Moore R, Collins S . Reporting of adverse effects in clinical trials should be improved: lessons from acute postoperative pain. J Pain Symptom Manage. 2000; 18(6):427-37. DOI: 10.1016/s0885-3924(99)00093-7. View

14.

MacLennan S, MacLennan A, Ryan P . Colorectal cancer and oestrogen replacement therapy. A meta-analysis of epidemiological studies. Med J Aust. 1995; 162(9):491-3. View

15.

Ross S, DiGEORGE A, Connelly J, Whiting G, McDonnell N . Safety of GM-CSF in patients with AIDS: a review of the literature. Pharmacotherapy. 1998; 18(6):1290-7. View

16.

Smith J, Green J, Berrington de Gonzalez A, Appleby P, Peto J, Plummer M . Cervical cancer and use of hormonal contraceptives: a systematic review. Lancet. 2003; 361(9364):1159-67. DOI: 10.1016/s0140-6736(03)12949-2. View

17.

Jones G, Riley M, Couper D, Dwyer T . Water fluoridation, bone mass and fracture: a quantitative overview of the literature. Aust N Z J Public Health. 1999; 23(1):34-40. DOI: 10.1111/j.1467-842x.1999.tb01202.x. View

18.

Martin G . Conflicting clinical trial data: a lesson from albumin. Crit Care. 2005; 9(6):649-50. PMC: 1414029. DOI: 10.1186/cc3931. View

19.

Levine M, Hamet P, Novosel S, Jolain B . A prospective comparison of four study designs used in assessing safety and effectiveness of drug therapy in hypertension management. Am J Hypertens. 1997; 10(11):1191-200. DOI: 10.1016/s0895-7061(97)00227-6. View

20.

Papanikolaou P, Ioannidis J . Availability of large-scale evidence on specific harms from systematic reviews of randomized trials. Am J Med. 2004; 117(8):582-9. DOI: 10.1016/j.amjmed.2004.04.026. View