Empirical Evidence of Design-related Bias in Studies of Diagnostic Tests
Overview
Affiliations
Context: The literature contains a large number of potential biases in the evaluation of diagnostic tests. Strict application of appropriate methodological criteria would invalidate the clinical application of most study results.
Objective: To empirically determine the quantitative effect of study design shortcomings on estimates of diagnostic accuracy.
Design And Setting: Observational study of the methodological features of 184 original studies evaluating 218 diagnostic tests. Meta-analyses on diagnostic tests were identified through a systematic search of the literature using MEDLINE, EMBASE, and DARE databases and the Cochrane Library (1996-1997). Associations between study characteristics and estimates of diagnostic accuracy were evaluated with a regression model.
Main Outcome Measures: Relative diagnostic odds ratio (RDOR), which compared the diagnostic odds ratios of studies of a given test that lacked a particular methodological feature with those without the corresponding shortcomings in design.
Results: Fifteen (6.8%) of 218 evaluations met all 8 criteria; 64 (30%) met 6 or more. Studies evaluating tests in a diseased population and a separate control group overestimated the diagnostic performance compared with studies that used a clinical population (RDOR, 3.0; 95% confidence interval [CI], 2.0-4.5). Studies in which different reference tests were used for positive and negative results of the test under study overestimated the diagnostic performance compared with studies using a single reference test for all patients (RDOR, 2.2; 95% CI, 1.5-3.3). Diagnostic performance was also overestimated when the reference test was interpreted with knowledge of the test result (RDOR, 1.3; 95% CI, 1.0-1.9), when no criteria for the test were described (RDOR, 1.7; 95% CI, 1.1-2.5), and when no description of the population under study was provided (RDOR, 1.4; 95% CI, 1.1-1.7).
Conclusion: These data provide empirical evidence that diagnostic studies with methodological shortcomings may overestimate the accuracy of a diagnostic test, particularly those including nonrepresentative patients or applying different reference standards.
A Seamless Design for the Combination of a Case-Control and a Cohort Diagnostic Accuracy Study.
Bibiza-Freiwald E, Vach W, Zapf A Stat Med. 2025; 44(6):e70016.
PMID: 40042437 PMC: 11881794. DOI: 10.1002/sim.70016.
Bernstein J, Cheng J, Pisani T, Sexton D, Whitaker R, Nova Estepan D J Allergy Clin Immunol Glob. 2025; 4(2):100401.
PMID: 39944300 PMC: 11814660. DOI: 10.1016/j.jacig.2025.100401.
Dixit R, Manikandan S, Prakash J, Biswal M, Mohapatra D, Gopalan N Front Microbiol. 2025; 15():1516921.
PMID: 39834374 PMC: 11743491. DOI: 10.3389/fmicb.2024.1516921.
Screening properties of the updated normative framework for the Italian MMSE in MCI and dementia.
Aiello E, Verde F, Curti B, De Luca G, Diana L, Sirtori M Neurol Sci. 2025; .
PMID: 39775365 DOI: 10.1007/s10072-024-07952-4.
Andreano A, Lepore V, Magnoni P, Milanese A, Fanizza C, Testa D Syst Rev. 2024; 13(1):313.
PMID: 39716260 PMC: 11668106. DOI: 10.1186/s13643-024-02717-8.