Implementing Multiple Imputations for Addressing Missing Data in Multireader Multicase Design Studies

Overview

Journal BMC Med Res Methodol

Publisher Biomed Central

Specialties General Medicine
Health Services

Date 2024 Sep 28

PMID 39333923

Authors

Zhemin Pan

Yingyi Qin

Wangyang Bai

Qian He

Xiaoping Yin

Jia He

Affiliations

Soon will be listed here.

Abstract

Background: In computer-aided diagnosis (CAD) studies utilizing multireader multicase (MRMC) designs, missing data might occur when there are instances of misinterpretation or oversight by the reader or problems with measurement techniques. Improper handling of these missing data can lead to bias. However, little research has been conducted on addressing the missing data issue within the MRMC framework.

Methods: We introduced a novel approach that integrates multiple imputation with MRMC analysis (MI-MRMC). An elaborate simulation study was conducted to compare the efficacy of our proposed approach with that of the traditional complete case analysis strategy within the MRMC design. Furthermore, we applied these approaches to a real MRMC design CAD study on aneurysm detection via head and neck CT angiograms to further validate their practicality.

Results: Compared with traditional complete case analysis, the simulation study demonstrated the MI-MRMC approach provides an almost unbiased estimate of diagnostic capability, alongside satisfactory performance in terms of statistical power and the type I error rate within the MRMC framework, even in small sample scenarios. In the real CAD study, the proposed MI-MRMC method further demonstrated strong performance in terms of both point estimates and confidence intervals compared with traditional complete case analysis.

Conclusion: Within MRMC design settings, the adoption of an MI-MRMC approach in the face of missing data can facilitate the attainment of unbiased and robust estimates of diagnostic capability.

References

Stahlmann K, Reitsma J, Zapf A . Missing values and inconclusive results in diagnostic studies - A scoping review of methods. Stat Methods Med Res. 2023; 32(9):1842-1855. PMC: 10540494. DOI: 10.1177/09622802231192954. View

Bartlett J, Seaman S, White I, Carpenter J . Multiple imputation of covariates by fully conditional specification: Accommodating the substantive model. Stat Methods Med Res. 2014; 24(4):462-87. PMC: 4513015. DOI: 10.1177/0962280214521348. View

Westreich D, Edwards J, Cole S, Platt R, Mumford S, Schisterman E . Imputation approaches for potential outcomes in causal inference. Int J Epidemiol. 2015; 44(5):1731-7. PMC: 4707196. DOI: 10.1093/ije/dyv135. View

Cheng W, Tang N . Smoothed empirical likelihood inference for ROC curve in the presence of missing biomarker values. Biom J. 2020; 62(4):1038-1059. DOI: 10.1002/bimj.201900121. View

Wang L, Wang H, Xia C, Wang Y, Tang Q, Li J . Toward standardized premarket evaluation of computer aided diagnosis/detection products: insights from FDA-approved products. Expert Rev Med Devices. 2020; 17(9):899-918. DOI: 10.1080/17434440.2020.1813566. View

Hillis S . A comparison of denominator degrees of freedom methods for multiple observer ROC analysis. Stat Med. 2006; 26(3):596-619. PMC: 1769324. DOI: 10.1002/sim.2532. View

Shinkins B, Thompson M, Mallett S, Perera R . Diagnostic accuracy studies: how to report and analyse inconclusive test results. BMJ. 2013; 346:f2778. DOI: 10.1136/bmj.f2778. View

Hickey G, Philipson P, Jorgensen A, Kolamunnage-Dona R . Joint modelling of time-to-event and multivariate longitudinal outcomes: recent developments and issues. BMC Med Res Methodol. 2016; 16(1):117. PMC: 5015261. DOI: 10.1186/s12874-016-0212-5. View

Hillis S, Berbaum K, Metz C . Recent developments in the Dorfman-Berbaum-Metz procedure for multireader ROC study analysis. Acad Radiol. 2008; 15(5):647-61. PMC: 2737462. DOI: 10.1016/j.acra.2007.12.015. View

10.

Pedersen A, Mikkelsen E, Cronin-Fenton D, Kristensen N, Pham T, Pedersen L . Missing data and multiple imputation in clinical epidemiological research. Clin Epidemiol. 2017; 9:157-166. PMC: 5358992. DOI: 10.2147/CLEP.S129785. View

11.

Hillis S . Simulation of unequal-variance binormal multireader ROC decision data: an extension of the Roe and Metz simulation model. Acad Radiol. 2012; 19(12):1518-28. PMC: 3532843. DOI: 10.1016/j.acra.2012.09.011. View

12.

Cohen J, Korevaar D, Altman D, Bruns D, Gatsonis C, Hooft L . STARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration. BMJ Open. 2017; 6(11):e012799. PMC: 5128957. DOI: 10.1136/bmjopen-2016-012799. View

13.

Kohn M, Carpenter C, Newman T . Understanding the direction of bias in studies of diagnostic test accuracy. Acad Emerg Med. 2013; 20(11):1194-206. DOI: 10.1111/acem.12255. View

14.

Gallas B, Chan H, DOrsi C, Dodd L, Giger M, Gur D . Evaluating imaging and computer-aided detection and diagnosis devices at the FDA. Acad Radiol. 2012; 19(4):463-77. PMC: 5557046. DOI: 10.1016/j.acra.2011.12.016. View

15.

Roe C, Metz C . Dorfman-Berbaum-Metz method for statistical analysis of multireader, multimodality receiver operating characteristic data: validation with computer simulation. Acad Radiol. 1997; 4(4):298-303. DOI: 10.1016/s1076-6332(97)80032-3. View

16.

Obuchowski N, Bullen J . Multireader Diagnostic Accuracy Imaging Studies: Fundamentals of Design and Analysis. Radiology. 2022; 303(1):26-34. DOI: 10.1148/radiol.211593. View

17.

Obuchowski N . Multireader, multimodality receiver operating characteristic curve studies: hypothesis testing and sample size estimation using an analysis of variance approach with dependent observations. Acad Radiol. 1995; 2 Suppl 1:S22-9; discussion S57-64, S70-1 pas. View

18.

Hillis S . Relationship between Roe and Metz simulation model for multireader diagnostic data and Obuchowski-Rockette model parameters. Stat Med. 2018; 37(13):2067-2093. PMC: 5980727. DOI: 10.1002/sim.7616. View

19.

Jakobsen J, Gluud C, Wetterslev J, Winkel P . When and how should multiple imputation be used for handling missing data in randomised clinical trials - a practical guide with flowcharts. BMC Med Res Methodol. 2017; 17(1):162. PMC: 5717805. DOI: 10.1186/s12874-017-0442-1. View

20.

Beam C, Layde P, Sullivan D . Variability in the interpretation of screening mammograms by US radiologists. Findings from a national sample. Arch Intern Med. 1996; 156(2):209-13. View