Mitigating Disputes Originated by Multiple Discordant Systematic Reviews and Meta-Analyses: A Survey of Methodologists and Clinicians

Overview

Journal Front Res Metr Anal

Date 2022 May 2

PMID 35494418

Authors

Livia Puljak

Elena Parmelli

Matteo Capobussi

Marien Gonzalez-Lorenzo

Alessandro Squizzato

Lorenzo Moja

Nicoletta Riva

Affiliations

Soon will be listed here.

Abstract

Background: Overlapping systematic reviews (SRs) are increasingly frequent in the medical literature. They can easily generate discordant evidence, as estimates of effect sizes and their interpretation might differ from one source to another.

Objective: To analyze how methodologists and clinicians make a decision when faced with discordant evidence formalized in structured tables.

Methods: We conducted a 16-item survey exploring how methodologists and clinicians would react when presented with multiple Summary of Findings (SoF) tables (generated using the GRADE tool) derived from 4 overlapping and discordant SRs and meta-analyses on thrombolytic therapy for intermediate-risk pulmonary embolism. SoF tables reported 4 different magnitudes of effects and overall certainty. Participants were asked to provide their recommendations regarding the intervention and the reasons behind their conclusion.

Results: Of the 80 invitees, 41 (51%) participated. The majority described themselves as "somewhat familiar" or experts with SoF tables. The majority recommended the therapy (pharmacological systemic thrombolysis), grading the recommendation as weak positive. Certainty of evidence and benefit-risk balance were the two criteria that prevailed in generating the recommendation. When faced with overlapping meta-analyses, the preferred approach was to use only high-quality SRs and exclude redundant SRs. Several participants suggested integrating the SoF tables with additional information, such as a more comprehensive evaluation of the risk of bias of systematic reviews (71%), heterogeneity/inconsistency (68%) and studies included within each SR (62%).

Conclusion: When faced with multiple controversial SR results, the type and completeness of reported information in SoF tables affect experts' ability to make recommendations. Developers of the SoF table should consider collating key information from overlapping and potentially discordant reviews.

Citing Articles

Appraisal of umbrella reviews on vaccines.

Bellomo R, Ioannidis J Commun Med (Lond). 2024; 4(1):250.

PMID: 39604580 PMC: 11603068. DOI: 10.1038/s43856-024-00679-5.

An overview of Frontiers in Research Metrics and Analytics.

Chen C, Chinchilla-Rodriguez Z, Zhang Y, Daniel B, Kajikawa Y, Wolfram D Front Res Metr Anal. 2024; 9:1420385.

PMID: 38832090 PMC: 11146195. DOI: 10.3389/frma.2024.1420385.

Definition, harms, and prevention of redundant systematic reviews.

Puljak L, Lund H Syst Rev. 2023; 12(1):63.

PMID: 37016459 PMC: 10071231. DOI: 10.1186/s13643-023-02191-8.

Evidence-based health policy in Germany: lack of communication and coordination between academia and health authorities?.

Kugler C, Perleth M, Mathes T, Goossen K, Pieper D Syst Rev. 2023; 12(1):36.

PMID: 36907893 PMC: 10010027. DOI: 10.1186/s13643-023-02204-6.

References

Lunny C, Brennan S, Reid J, McDonald S, McKenzie J . Overviews of reviews incompletely report methods for handling overlapping, discordant, and problematic data. J Clin Epidemiol. 2019; 118:69-85. DOI: 10.1016/j.jclinepi.2019.09.025. View

Gao G, Yang P, Liu M, Ding M, Liu G, Tong Y . Thrombolysis for acute intermediate-risk pulmonary embolism: A meta-analysis. Thromb Res. 2015; 136(5):932-7. DOI: 10.1016/j.thromres.2015.09.012. View

Munn Z, Stern C, Aromataris E, Lockwood C, Jordan Z . What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences. BMC Med Res Methodol. 2018; 18(1):5. PMC: 5761190. DOI: 10.1186/s12874-017-0468-4. View

Augustin G, Boric M, Barcot O, Puljak L . Discordant outcomes of laparoscopic versus open appendectomy for suspected appendicitis during pregnancy in published meta-analyses: an overview of systematic reviews. Surg Endosc. 2020; 34(10):4245-4256. DOI: 10.1007/s00464-020-07674-6. View

Ioannidis J . Next-generation systematic reviews: prospective meta-analysis, individual-level data, networks and umbrella reviews. Br J Sports Med. 2017; 51(20):1456-1458. DOI: 10.1136/bjsports-2017-097621. View

Jadad A, Cook D, Browman G . A guide to interpreting discordant systematic reviews. CMAJ. 1997; 156(10):1411-6. PMC: 1227410. View

Bobbio M, Demichelis B, Giustetto G . Completeness of reporting trial results: effect on physicians' willingness to prescribe. Lancet. 1994; 343(8907):1209-11. DOI: 10.1016/s0140-6736(94)92407-4. View

Hao Q, Dong B, Yue J, Wu T, Liu G . Thrombolytic therapy for pulmonary embolism. Cochrane Database Syst Rev. 2018; 12:CD004437. PMC: 6516871. DOI: 10.1002/14651858.CD004437.pub5. View

Alper B, Haynes R . EBHC pyramid 5.0 for accessing preappraised evidence and guidance. Evid Based Med. 2016; 21(4):123-5. DOI: 10.1136/ebmed-2016-110447. View

10.

Riva N, Puljak L, Moja L, Ageno W, Schunemann H, Magrini N . Multiple overlapping systematic reviews facilitate the origin of disputes: the case of thrombolytic therapy for pulmonary embolism. J Clin Epidemiol. 2017; 97:1-13. DOI: 10.1016/j.jclinepi.2017.11.012. View

11.

Guyatt G, Oxman A, Akl E, Kunz R, Vist G, Brozek J . GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011; 64(4):383-94. DOI: 10.1016/j.jclinepi.2010.04.026. View

12.

Krnic Martinic M, Pieper D, Glatt A, Puljak L . Definition of a systematic review used in overviews of systematic reviews, meta-epidemiological studies and textbooks. BMC Med Res Methodol. 2019; 19(1):203. PMC: 6829801. DOI: 10.1186/s12874-019-0855-0. View

13.

. Evidence-based medicine. A new approach to teaching the practice of medicine. JAMA. 1992; 268(17):2420-5. DOI: 10.1001/jama.1992.03490170092032. View

14.

Greenhalgh T, Howick J, Maskrey N . Evidence based medicine: a movement in crisis?. BMJ. 2014; 348:g3725. PMC: 4056639. DOI: 10.1136/bmj.g3725. View

15.

Dettori J, Norvell D . Discordant Systematic Reviews: Which to Believe?. Global Spine J. 2020; 10(2):237-239. PMC: 7076591. DOI: 10.1177/2192568219899945. View

16.

Puljak L . If there is only one author or only one database was searched, a study should not be called a systematic review. J Clin Epidemiol. 2017; 91:4-5. DOI: 10.1016/j.jclinepi.2017.08.002. View

17.

Chatterjee S, Chakraborty A, Weinberg I, Kadakia M, Wilensky R, Sardar P . Thrombolysis for pulmonary embolism and risk of all-cause mortality, major bleeding, and intracranial hemorrhage: a meta-analysis. JAMA. 2014; 311(23):2414-21. DOI: 10.1001/jama.2014.5990. View

18.

Hoffmann F, Allers K, Rombey T, Helbach J, Hoffmann A, Mathes T . Nearly 80 systematic reviews were published each day: Observational study on trends in epidemiology and reporting over the years 2000-2019. J Clin Epidemiol. 2021; 138:1-11. DOI: 10.1016/j.jclinepi.2021.05.022. View

19.

Lucenteforte E, Moja L, Pecoraro V, Conti A, Conti A, Crudeli E . Discordances originated by multiple meta-analyses on interventions for myocardial infarction: a systematic review. J Clin Epidemiol. 2014; 68(3):246-56. DOI: 10.1016/j.jclinepi.2014.11.004. View

20.

Ren C, Chen H . Thrombolysis versus anticoagulation for the initial treatment of moderate pulmonary embolism: a meta-analysis of randomized controlled trials. Respir Care. 2014; 59(12):1880-7. DOI: 10.4187/respcare.03197. View