Evaluating Imaging and Computer-aided Detection and Diagnosis Devices at the FDA

Overview

Journal Acad Radiol

Specialty Radiology

Date 2012 Feb 7

PMID 22306064

Citations 33

Authors

Brandon D Gallas

Heang-Ping Chan

Carl J DOrsi

Lori E Dodd

Maryellen L Giger

David Gur

Elizabeth A Krupinski

Charles E Metz

Kyle J Myers

Nancy A Obuchowski

Berkman Sahiner

Alicia Y Toledano

Margarita L Zuley

Affiliations

Soon will be listed here.

Abstract

This report summarizes the Joint FDA-MIPS Workshop on Methods for the Evaluation of Imaging and Computer-Assist Devices. The purpose of the workshop was to gather information on the current state of the science and facilitate consensus development on statistical methods and study designs for the evaluation of imaging devices to support US Food and Drug Administration submissions. Additionally, participants expected to identify gaps in knowledge and unmet needs that should be addressed in future research. This summary is intended to document the topics that were discussed at the meeting and disseminate the lessons that have been learned through past studies of imaging and computer-aided detection and diagnosis device performance.

Citing Articles

G4 & the balanced metric family - a novel approach to solving binary classification problems in medical device validation & verification studies.

Marra A BioData Min. 2024; 17(1):43.

PMID: 39444008 PMC: 11515465. DOI: 10.1186/s13040-024-00402-z.

Implementing multiple imputations for addressing missing data in multireader multicase design studies.

Pan Z, Qin Y, Bai W, He Q, Yin X, He J BMC Med Res Methodol. 2024; 24(1):217.

PMID: 39333923 PMC: 11428558. DOI: 10.1186/s12874-024-02321-3.

Assistive AI in Lung Cancer Screening: A Retrospective Multinational Study in the United States and Japan.

Kiraly A, Cunningham C, Najafi R, Nabulsi Z, Yang J, Lau C Radiol Artif Intell. 2024; 6(3):e230079.

PMID: 38477661 PMC: 11140517. DOI: 10.1148/ryai.230079.

End-to-end deep learning method for predicting hormonal treatment response in women with atypical endometrial hyperplasia or endometrial cancer.

Kahaki S, Hagemann I, Cha K, Trindade C, Petrick N, Kostelecky N J Med Imaging (Bellingham). 2024; 11(1):017502.

PMID: 38370423 PMC: 10868592. DOI: 10.1117/1.JMI.11.1.017502.

Analysis of cellularity in H&E-stained rat bone marrow tissue via deep learning.

Shiffman S, Rios Piedra E, Adedeji A, Ruff C, Andrews R, Katavolos P J Pathol Inform. 2023; 14:100333.

PMID: 37743975 PMC: 10514468. DOI: 10.1016/j.jpi.2023.100333.

References

Beiden S, Wagner R, Doi K, Nishikawa R, Freedman M, Lo S . Independent versus sequential reading in ROC studies of computer-assist modalities: analysis of components of variance. Acad Radiol. 2002; 9(9):1036-43. DOI: 10.1016/s1076-6332(03)80479-8. View

Gur D, Sumkin J, Hardesty L, Clearfield R, Cohen C, Ganott M . Recall and detection rates in screening mammography. Cancer. 2004; 100(8):1590-4. DOI: 10.1002/cncr.20053. View

Pinsky P, Gallas B . Enriched designs for assessing discriminatory performance--analysis of bias and variance. Stat Med. 2011; 31(6):501-15. DOI: 10.1002/sim.4432. View

Obuchowski N . Reducing the number of reader interpretations in MRMC studies. Acad Radiol. 2009; 16(2):209-17. DOI: 10.1016/j.acra.2008.05.014. View

Beam C, Layde P, Sullivan D . Variability in the interpretation of screening mammograms by US radiologists. Findings from a national sample. Arch Intern Med. 1996; 156(2):209-13. View

Starr S, Metz C, Lusted L, Goodenough D . Visual detection and localization of radiographic images. Radiology. 1975; 116(3):533-8. DOI: 10.1148/116.3.533. View

Yablon C, Wu J, Slanetz P, Eisenberg R . A report on the current status of grand rounds in radiology residency programs in the United States. Acad Radiol. 2011; 18(12):1593-7. DOI: 10.1016/j.acra.2011.08.015. View

Gur D, Rockette H, Armfield D, Blachar A, Bogan J, Brancatelli G . Prevalence effect in a laboratory environment. Radiology. 2003; 228(1):10-4. DOI: 10.1148/radiol.2281020709. View

Dorfman D, Berbaum K, Metz C . Receiver operating characteristic rating analysis. Generalization to the population of readers and patients with the jackknife method. Invest Radiol. 1992; 27(9):723-31. View

10.

Pesce L, Metz C, Berbaum K . On the convexity of ROC curves estimated from radiological test results. Acad Radiol. 2010; 17(8):960-968.e4. PMC: 2897827. DOI: 10.1016/j.acra.2010.04.001. View

11.

Chan H, Sahiner B, Wagner R, Petrick N . Classifier design for computer-aided diagnosis: effects of finite sample size on the mean performance of classical and neural network classifiers. Med Phys. 2000; 26(12):2654-68. DOI: 10.1118/1.598805. View

12.

McCarney R, Warner J, Iliffe S, Haselen R, Griffin M, Fisher P . The Hawthorne Effect: a randomised, controlled trial. BMC Med Res Methodol. 2007; 7:30. PMC: 1936999. DOI: 10.1186/1471-2288-7-30. View

13.

Kobayashi T, Xu X, Macmahon H, Metz C, Doi K . Effect of a computer-aided diagnosis scheme on radiologists' performance in detection of lung nodules on radiographs. Radiology. 1996; 199(3):843-8. DOI: 10.1148/radiology.199.3.8638015. View

14.

Gallas B . One-shot estimate of MRMC variance: AUC. Acad Radiol. 2006; 13(3):353-62. DOI: 10.1016/j.acra.2005.11.030. View

15.

Chakraborty D . Analysis of location specific observer performance data: validated extensions of the jackknife free-response (JAFROC) method. Acad Radiol. 2006; 13(10):1187-93. DOI: 10.1016/j.acra.2006.06.016. View

16.

Simon R, Radmacher M, Dobbin K, McShane L . Pitfalls in the use of DNA microarray data for diagnostic and prognostic classification. J Natl Cancer Inst. 2003; 95(1):14-8. DOI: 10.1093/jnci/95.1.14. View

17.

Kupinski M, Giger M . Feature selection with limited datasets. Med Phys. 1999; 26(10):2176-82. DOI: 10.1118/1.598821. View

18.

Edwards D, Kupinski M, Metz C, Nishikawa R . Maximum likelihood fitting of FROC curves under an initial-detection-and-candidate-analysis model. Med Phys. 2003; 29(12):2861-70. DOI: 10.1118/1.1524631. View

19.

Shaw L, Wilson P, Hachamovitch R, Hendel R, Borges-Neto S, Berman D . Improved near-term coronary artery disease risk classification with gated stress myocardial perfusion SPECT. JACC Cardiovasc Imaging. 2010; 3(11):1139-48. DOI: 10.1016/j.jcmg.2010.09.008. View

20.

Jiang Y, Metz C, Nishikawa R . A receiver operating characteristic partial area index for highly sensitive diagnostic tests. Radiology. 1996; 201(3):745-50. DOI: 10.1148/radiology.201.3.8939225. View