Identifying Outcome Reporting Bias in Randomised Trials on PubMed: Review of Publications and Survey of Authors

Overview

Journal BMJ

Specialty General Medicine

Date 2005 Feb 1

PMID 15681569

Citations 187

Authors

An-Wen Chan

Douglas G Altman

Affiliations

Soon will be listed here.

Abstract

Objective: To examine the extent and nature of outcome reporting bias in a broad cohort of published randomised trials.

Design: Retrospective review of publications and follow up survey of authors. Cohort All journal articles of randomised trials indexed in PubMed whose primary publication appeared in December 2000.

Main Outcome Measures: Prevalence of incompletely reported outcomes per trial; reasons for not reporting outcomes; association between completeness of reporting and statistical significance.

Results: 519 trials with 553 publications and 10,557 outcomes were identified. Survey responders (response rate 69%) provided information on unreported outcomes but were often unreliable--for 32% of those who denied the existence of such outcomes there was evidence to the contrary in their publications. On average, over 20% of the outcomes measured in a parallel group trial were incompletely reported. Within a trial, such outcomes had a higher odds of being statistically non-significant compared with fully reported outcomes (odds ratio 2.0 (95% confidence interval 1.6 to 2.7) for efficacy outcomes; 1.9 (1.1 to 3.5) for harm outcomes). The most commonly reported reasons for omitting efficacy outcomes included space constraints, lack of clinical importance, and lack of statistical significance.

Conclusions: Incomplete reporting of outcomes within published articles of randomised trials is common and is associated with statistical non-significance. The medical literature therefore represents a selective and biased subset of study outcomes, and trial protocols should be made publicly available.

Citing Articles

The role of QI collaboratives in neonatology.

Bapat R, Pearlman S J Perinatol. 2024; .

PMID: 39384615 DOI: 10.1038/s41372-024-02124-w.

Primary outcome reporting in clinical trials for older adults with depression.

Rodrigues M, Oprea A, Johnson K, Dufort A, Sanger N, Ghiassi P BJPsych Open. 2024; 10(2):e60.

PMID: 38450491 PMC: 10951853. DOI: 10.1192/bjo.2023.650.

Meta-research on reporting guidelines for artificial intelligence: are authors and reviewers encouraged enough in radiology, nuclear medicine, and medical imaging journals?.

Kocak B, Keles A, Kose F Diagn Interv Radiol. 2024; 30(5):291-298.

PMID: 38375627 PMC: 11590734. DOI: 10.4274/dir.2024.232604.

Oral Toxicities Associated with Immune Checkpoint Inhibitors: Meta-Analyses of Clinical Trials.

Srivastava A, Nogueras-Gonzalez G, Geng Y, Singh J, Myers J, Li Y J Immunother Precis Oncol. 2024; 7(1):24-40.

PMID: 38327757 PMC: 10846637. DOI: 10.36401/JIPO-23-14.

Unveiling Bias: Examining the Influence of Positive Results on Ergogenic Aids in Published Sports Science Studies.

Marocolo M, Mota G, Rodrigues A, Leite R, Hohl R, Paixao R Sports Med Int Open. 2024; 8:a21816798.

PMID: 38312926 PMC: 10832574. DOI: 10.1055/a-2181-6798.

References

Chan A, Altman D . Epidemiology and reporting of randomised trials published in PubMed journals. Lancet. 2005; 365(9465):1159-62. DOI: 10.1016/S0140-6736(05)71879-1. View

Moher D, Schulz K, Altman D . The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet. 2001; 357(9263):1191-4. View

Hahn S, Williamson P, Hutton J . Investigation of within-study selective reporting in clinical research: follow-up of applications submitted to a local research ethics committee. J Eval Clin Pract. 2002; 8(3):353-9. DOI: 10.1046/j.1365-2753.2002.00314.x. View

Staessen J, Bianchi G . Registration of trials and protocols. Lancet. 2003; 362(9389):1009-10. DOI: 10.1016/S0140-6736(03)14448-0. View

Elbourne D, Altman D, Higgins J, Curtin F, Worthington H, Vail A . Meta-analyses involving cross-over trials: methodological issues. Int J Epidemiol. 2002; 31(1):140-9. DOI: 10.1093/ije/31.1.140. View

Tannock I . False-positive results in clinical trials: multiple significance tests and the problem of unreported comparisons. J Natl Cancer Inst. 1996; 88(3-4):206-7. DOI: 10.1093/jnci/88.3-4.206. View

Johansen H, Gotzsche P . Problems in the design and reporting of trials of antifungal agents encountered during meta-analysis. JAMA. 1999; 282(18):1752-9. DOI: 10.1001/jama.282.18.1752. View

Thompson S, Sharp S . Explaining heterogeneity in meta-analysis: a comparison of methods. Stat Med. 1999; 18(20):2693-708. DOI: 10.1002/(sici)1097-0258(19991030)18:20<2693::aid-sim235>3.0.co;2-v. View

Chan A, Hrobjartsson A, Haahr M, Gotzsche P, Altman D . Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004; 291(20):2457-65. DOI: 10.1001/jama.291.20.2457. View

10.

Chan A, Krleza-Jeric K, Schmid I, Altman D . Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ. 2004; 171(7):735-40. PMC: 517858. DOI: 10.1503/cmaj.1041086. View

11.

Chalmers I, Altman D . How can medical journals help prevent poor medical research? Some opportunities presented by electronic publishing. Lancet. 1999; 353(9151):490-3. DOI: 10.1016/S0140-6736(98)07618-1. View

12.

Ioannidis J, Evans S, Gotzsche P, ONeill R, Altman D, Schulz K . Better reporting of harms in randomized trials: an extension of the CONSORT statement. Ann Intern Med. 2004; 141(10):781-8. DOI: 10.7326/0003-4819-141-10-200411160-00009. View

13.

Hampton J, Julian D . Role of the pharmaceutical industry in major clinical trials. Lancet. 1987; 2(8570):1258-9. DOI: 10.1016/s0140-6736(87)91863-0. View

14.

Streiner D, Joffe R . The adequacy of reporting randomized, controlled trials in the evaluation of antidepressants. Can J Psychiatry. 1998; 43(10):1026-30. DOI: 10.1177/070674379804301008. View

15.

Robinson K, Dickersin K . Development of a highly sensitive search strategy for the retrieval of reports of controlled trials using PubMed. Int J Epidemiol. 2002; 31(1):150-3. DOI: 10.1093/ije/31.1.150. View

16.

Felson D . Bias in meta-analytic research. J Clin Epidemiol. 1992; 45(8):885-92. DOI: 10.1016/0895-4356(92)90072-u. View

17.

Song F, Eastwood A, Gilbody S, Duley L, Sutton A . Publication and related biases. Health Technol Assess. 2000; 4(10):1-115. View

18.

Blumenthal D, CAMPBELL E, Anderson M, Causino N, Louis K . Withholding research results in academic life science. Evidence from a national survey of faculty. JAMA. 1997; 277(15):1224-8. View