Major Mistakes or Errors in the Use of Trial Sequential Analysis in Systematic Reviews or Meta-analyses - the METSA Systematic Review

Overview

Journal BMC Med Res Methodol

Publisher Biomed Central

Specialties General Medicine
Health Services

Date 2024 Sep 9

PMID 39251912

Authors

Christian Gunge Riberholt

Markus Harboe Olsen

Joachim Birch Milan

Sigurlaug Hanna Haflidadottir

Jeppe Houmann Svanholm

Elisabeth Buck Pedersen

Charles Chin Han Lew

Mark Aninakwah Asante

Johanne Pereira Ribeiro

Vibeke Wagner

Buddheera W M B Kumburegama

Zheng-Yii Lee

Julie Perrine Schaug

Christina Madsen

Christian Gluud

Affiliations

Soon will be listed here.

Abstract

Background: Systematic reviews and data synthesis of randomised clinical trials play a crucial role in clinical practice, research, and health policy. Trial sequential analysis can be used in systematic reviews to control type I and type II errors, but methodological errors including lack of protocols and transparency are cause for concern. We assessed the reporting of trial sequential analysis.

Methods: We searched Medline and the Cochrane Database of Systematic Reviews from 1 January 2018 to 31 December 2021 for systematic reviews and meta-analysis reports that include a trial sequential analysis. Only studies with at least two randomised clinical trials analysed in a forest plot and a trial sequential analysis were included. Two independent investigators assessed the studies. We evaluated protocolisation, reporting, and interpretation of the analyses, including their effect on any GRADE evaluation of imprecision.

Results: We included 270 systematic reviews and 274 meta-analysis reports and extracted data from 624 trial sequential analyses. Only 134/270 (50%) systematic reviews planned the trial sequential analysis in the protocol. For analyses on dichotomous outcomes, the proportion of events in the control group was missing in 181/439 (41%), relative risk reduction in 105/439 (24%), alpha in 30/439 (7%), beta in 128/439 (29%), and heterogeneity in 232/439 (53%). For analyses on continuous outcomes, the minimally relevant difference was missing in 125/185 (68%), variance (or standard deviation) in 144/185 (78%), alpha in 23/185 (12%), beta in 63/185 (34%), and heterogeneity in 105/185 (57%). Graphical illustration of the trial sequential analysis was present in 93% of the analyses, however, the Z-curve was wrongly displayed in 135/624 (22%) and 227/624 (36%) did not include futility boundaries. The overall transparency of all 624 analyses was very poor in 236 (38%) and poor in 173 (28%).

Conclusions: The majority of trial sequential analyses are not transparent when preparing or presenting the required parameters, partly due to missing or poorly conducted protocols. This hampers interpretation, reproducibility, and validity.

Study Registration: PROSPERO CRD42021273811.

References

Harris P, Taylor R, Thielke R, Payne J, Gonzalez N, Conde J . Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2008; 42(2):377-81. PMC: 2700030. DOI: 10.1016/j.jbi.2008.08.010. View

Moher D, Tetzlaff J, Tricco A, Sampson M, Altman D . Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007; 4(3):e78. PMC: 1831728. DOI: 10.1371/journal.pmed.0040078. View

Glasziou P, Altman D, Bossuyt P, Boutron I, Clarke M, Julious S . Reducing waste from incomplete or unusable reports of biomedical research. Lancet. 2014; 383(9913):267-76. DOI: 10.1016/S0140-6736(13)62228-X. View

Korang S, von Rohden E, Veroniki A, Ong G, Ngalamika O, Siddiqui F . Vaccines to prevent COVID-19: A living systematic review with Trial Sequential Analysis and network meta-analysis of randomized clinical trials. PLoS One. 2022; 17(1):e0260733. PMC: 8782520. DOI: 10.1371/journal.pone.0260733. View

Zhao X, Jiang H, Yin J, Liu H, Zhu R, Mei S . Changing trends in clinical research literature on PubMed database from 1991 to 2020. Eur J Med Res. 2022; 27(1):95. PMC: 9208110. DOI: 10.1186/s40001-022-00717-9. View

Windsor B, Popovich I, Jordan V, Showell M, Shea B, Farquhar C . Methodological quality of systematic reviews in subfertility: a comparison of Cochrane and non-Cochrane systematic reviews in assisted reproductive technologies. Hum Reprod. 2012; 27(12):3460-6. DOI: 10.1093/humrep/des342. View

Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M . Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015; 4:1. PMC: 4320440. DOI: 10.1186/2046-4053-4-1. View

Salman R, Beller E, Kagan J, Hemminki E, Phillips R, Savulescu J . Increasing value and reducing waste in biomedical research regulation and management. Lancet. 2014; 383(9912):176-85. PMC: 3952153. DOI: 10.1016/S0140-6736(13)62297-7. View

Garattini S, Jakobsen J, Wetterslev J, Bertele V, Banzi R, Rath A . Evidence-based clinical practice: Overview of threats to the validity of evidence and how to minimise them. Eur J Intern Med. 2016; 32:13-21. DOI: 10.1016/j.ejim.2016.03.020. View

10.

Nguyen V, Jung K, Gupta V . Examining data visualization pitfalls in scientific publications. Vis Comput Ind Biomed Art. 2021; 4(1):27. PMC: 8556474. DOI: 10.1186/s42492-021-00092-y. View

11.

Wetterslev J, Thorlund K, Brok J, Gluud C . Estimating required information size by quantifying diversity in random-effects model meta-analyses. BMC Med Res Methodol. 2010; 9:86. PMC: 2809074. DOI: 10.1186/1471-2288-9-86. View

12.

Wetterslev J, Jakobsen J, Gluud C . Trial Sequential Analysis in systematic reviews with meta-analysis. BMC Med Res Methodol. 2017; 17(1):39. PMC: 5397700. DOI: 10.1186/s12874-017-0315-7. View

13.

Useem J, Brennan A, LaValley M, Vickery M, Ameli O, Reinen N . Systematic Differences between Cochrane and Non-Cochrane Meta-Analyses on the Same Topic: A Matched Pair Analysis. PLoS One. 2015; 10(12):e0144980. PMC: 4686011. DOI: 10.1371/journal.pone.0144980. View

14.

Sidebotham D, Popovich I, Lumley T . A Bayesian analysis of mortality outcomes in multicentre clinical trials in critical care. Br J Anaesth. 2021; 127(3):487-494. DOI: 10.1016/j.bja.2021.06.026. View

15.

Turner R, Bird S, Higgins J . The impact of study size on meta-analyses: examination of underpowered studies in Cochrane reviews. PLoS One. 2013; 8(3):e59202. PMC: 3609745. DOI: 10.1371/journal.pone.0059202. View

16.

Wetterslev J, Thorlund K, Brok J, Gluud C . Trial sequential analysis may establish when firm evidence is reached in cumulative meta-analysis. J Clin Epidemiol. 2007; 61(1):64-75. DOI: 10.1016/j.jclinepi.2007.03.013. View

17.

Schulz K, Altman D, Moher D . CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ. 2010; 340:c332. PMC: 2844940. DOI: 10.1136/bmj.c332. View

18.

Claire R, Gluud C, Berlin I, Coleman T, Leonardi-Bee J . Using Trial Sequential Analysis for estimating the sample sizes of further trials: example using smoking cessation intervention. BMC Med Res Methodol. 2020; 20(1):284. PMC: 7702700. DOI: 10.1186/s12874-020-01169-7. View

19.

Gluud C . Testosterone and alcoholic cirrhosis. Epidemiologic, pathophysiologic and therapeutic studies in men. Dan Med Bull. 1988; 35(6):564-75. View

20.

Ioannidis J . The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses. Milbank Q. 2016; 94(3):485-514. PMC: 5020151. DOI: 10.1111/1468-0009.12210. View