Significance Testing As Perverse Probabilistic Reasoning

Overview

Journal BMC Med

Publisher Biomed Central

Specialty General Medicine

Date 2011 Mar 2

PMID 21356064

Citations 7

Authors

M Brandon Westover

Kenneth D Westover

Matt T Bianchi

Affiliations

Soon will be listed here.

Abstract

Truth claims in the medical literature rely heavily on statistical significance testing. Unfortunately, most physicians misunderstand the underlying probabilistic logic of significance tests and consequently often misinterpret their results. This near-universal misunderstanding is highlighted by means of a simple quiz which we administered to 246 physicians at two major academic hospitals, on which the proportion of incorrect responses exceeded 90%. A solid understanding of the fundamental concepts of probability theory is becoming essential to the rational interpretation of medical information. This essay provides a technically sound review of these concepts that is accessible to a medical audience. We also briefly review the debate in the cognitive sciences regarding physicians' aptitude for probabilistic inference.

Citing Articles

Severely malnourished children with a low weight-for-height have a higher mortality than those with a low mid-upper-arm-circumference: I. Empirical data demonstrates Simpson's paradox.

Grellety E, Golden M Nutr J. 2018; 17(1):79.

PMID: 30217205 PMC: 6138885. DOI: 10.1186/s12937-018-0384-4.

Rule of 100: a litmus test for informationless diagnostic tests.

Westover A, Shapiro D, Westover M, Bianchi M Postgrad Med J. 2018; 94(1112):364-366.

PMID: 29514996 PMC: 6771257. DOI: 10.1136/postgradmedj-2018-135555.

Big data in sleep medicine: prospects and pitfalls in phenotyping.

Bianchi M, Russo K, Gabbidon H, Smith T, Goparaju B, Westover M Nat Sci Sleep. 2017; 9:11-29.

PMID: 28243157 PMC: 5317347. DOI: 10.2147/NSS.S130141.

Predicting Progression from Mild Cognitive Impairment to Alzheimer's Dementia Using Clinical, MRI, and Plasma Biomarkers via Probabilistic Pattern Classification.

Korolev I, Symonds L, Bozoki A PLoS One. 2016; 11(2):e0138866.

PMID: 26901338 PMC: 4762666. DOI: 10.1371/journal.pone.0138866.

From P-Values to Objective Probabilities in Assessing Medical Treatments.

Kault D, Kault S PLoS One. 2015; 10(11):e0142132.

PMID: 26600468 PMC: 4658034. DOI: 10.1371/journal.pone.0142132.

References

Ioannidis J . Microarrays and molecular research: noise discovery?. Lancet. 2005; 365(9458):454-5. DOI: 10.1016/S0140-6736(05)17878-7. View

Ioannidis J . Why most published research findings are false. PLoS Med. 2005; 2(8):e124. PMC: 1182327. DOI: 10.1371/journal.pmed.0020124. View

Phelps M, Levitt M . Pretest probability estimates: a pitfall to the clinical utility of evidence-based medicine?. Acad Emerg Med. 2004; 11(6):692-4. View

Oaksford M, Chater N . Précis of bayesian rationality: The probabilistic approach to human reasoning. Behav Brain Sci. 2009; 32(1):69-84. DOI: 10.1017/S0140525X09000284. View

Vandenbroucke J . When are observational studies as credible as randomised trials?. Lancet. 2004; 363(9422):1728-31. DOI: 10.1016/S0140-6736(04)16261-2. View

Dixon P . The p-value fallacy and how to avoid it. Can J Exp Psychol. 2003; 57(3):189-202. DOI: 10.1037/h0087425. View

Kruschke J . What to believe: Bayesian methods for data analysis. Trends Cogn Sci. 2010; 14(7):293-300. DOI: 10.1016/j.tics.2010.05.001. View

Nickerson R . Null hypothesis significance testing: a review of an old and continuing controversy. Psychol Methods. 2000; 5(2):241-301. DOI: 10.1037/1082-989x.5.2.241. View

Goodman S . Introduction to Bayesian methods I: measuring the strength of evidence. Clin Trials. 2005; 2(4):282-90. DOI: 10.1191/1740774505cn098oa. View

10.

Wagenmakers E . A practical solution to the pervasive problems of p values. Psychon Bull Rev. 2007; 14(5):779-804. DOI: 10.3758/bf03194105. View

11.

Casscells W, SCHOENBERGER A, Graboys T . Interpretation by physicians of clinical laboratory results. N Engl J Med. 1978; 299(18):999-1001. DOI: 10.1056/NEJM197811022991808. View

12.

Windish D, Huot S, Green M . Medicine residents' understanding of the biostatistics and results in the medical literature. JAMA. 2007; 298(9):1010-22. DOI: 10.1001/jama.298.9.1010. View

13.

Oaksford M, Chater N . The probabilistic approach to human reasoning. Trends Cogn Sci. 2001; 5(8):349-357. DOI: 10.1016/s1364-6613(00)01699-5. View

14.

Clarke M, Alderson P, Chalmers I . Discussion sections in reports of controlled trials published in general medical journals. JAMA. 2002; 287(21):2799-801. DOI: 10.1001/jama.287.21.2799. View

15.

Browner W, Newman T . Are all significant P values created equal? The analogy between diagnostic tests and clinical research. JAMA. 1987; 257(18):2459-63. View

16.

Forrester M, Pettitt A, Gibson G . Bayesian inference of hospital-acquired infectious diseases and control measures given imperfect surveillance data. Biostatistics. 2006; 8(2):383-401. DOI: 10.1093/biostatistics/kxl017. View

17.

Horton N, Switzer S . Statistical methods in the journal. N Engl J Med. 2005; 353(18):1977-9. DOI: 10.1056/NEJM200511033531823. View

18.

Goodman S . Toward evidence-based medical statistics. 2: The Bayes factor. Ann Intern Med. 1999; 130(12):1005-13. DOI: 10.7326/0003-4819-130-12-199906150-00019. View

19.

Ioannidis J . Contradicted and initially stronger effects in highly cited clinical research. JAMA. 2005; 294(2):218-28. DOI: 10.1001/jama.294.2.218. View

20.

Spiegelhalter D, Myles J, Jones D, Abrams K . Bayesian methods in health technology assessment: a review. Health Technol Assess. 2001; 4(38):1-130. View