A Comprehensive Review of Performance Metrics for Computer-Aided Detection Systems

Overview

Journal Bioengineering (Basel)

Date 2024 Nov 27

PMID 39593823

Authors

Doohyun Park

Affiliations

Soon will be listed here.

Abstract

This paper aims to provide a structured analysis of the performance metrics used in computer-aided detection (CAD) systems, specifically focusing on lung nodule detection in computed tomography (CT) images. By examining key metrics along with their respective strengths and limitations, this study offers guidelines to assist in selecting appropriate metrics. Evaluation methods for CAD systems for lung nodule detection are primarily categorized into per-scan and per-nodule approaches. For per-scan analysis, a key metric is the area under the receiver operating characteristic (ROC) curve (AUROC), which evaluates the ability of the system to distinguish between scans with and without nodules. For per-nodule analysis, the nodule-level sensitivity at fixed false positives per scan is often used, supplemented by the free-response receiver operating characteristic (FROC) curve and the competition performance metric (CPM). However, the CPM does not provide normalized scores because it theoretically ranges from zero to infinity and largely varies depending on the characteristics of the data. To address the advantages and limitations of ROC and FROC curves, an alternative FROC (AFROC) was introduced to combine the strengths of both per-scan and per-nodule analyses. This paper discusses the principles of each metric and their relative strengths, providing insights into their clinical implications and practical utility.

References

Li L, Liu Z, Huang H, Lin M, Luo D . Evaluating the performance of a deep learning-based computer-aided diagnosis (DL-CAD) system for detecting and characterizing lung nodules: Comparison with the performance of double reading by radiologists. Thorac Cancer. 2018; 10(2):183-192. PMC: 6360226. DOI: 10.1111/1759-7714.12931. View

Hsu H, Ko K, Chou Y, Wu Y, Chiu S, Chang C . Performance and reading time of lung nodule identification on multidetector CT with or without an artificial intelligence-powered computer-aided detection system. Clin Radiol. 2021; 76(8):626.e23-626.e32. DOI: 10.1016/j.crad.2021.04.006. View

Jin W, Li X, Fatehi M, Hamarneh G . Guidelines and evaluation of clinical explainable AI in medical image analysis. Med Image Anal. 2022; 84:102684. DOI: 10.1016/j.media.2022.102684. View

Park S, Lee S, Kim W, Park H, Jung K, Do K . Computer-aided Detection of Subsolid Nodules at Chest CT: Improved Performance with Deep Learning-based CT Section Thickness Reduction. Radiology. 2021; 299(1):211-219. DOI: 10.1148/radiol.2021203387. View

Katase S, Ichinose A, Hayashi M, Watanabe M, Chin K, Takeshita Y . Development and performance evaluation of a deep learning lung nodule detection system. BMC Med Imaging. 2022; 22(1):203. PMC: 9682774. DOI: 10.1186/s12880-022-00938-8. View

Al Mohammad B, Brennan P, Mello-Thoms C . A review of lung cancer screening and the role of computer-aided detection. Clin Radiol. 2017; 72(6):433-442. DOI: 10.1016/j.crad.2017.01.002. View

Singh S, Pinsky P, Fineberg N, Gierada D, Garg K, Sun Y . Evaluation of reader variability in the interpretation of follow-up CT scans at lung cancer screening. Radiology. 2011; 259(1):263-70. PMC: 3064819. DOI: 10.1148/radiol.10101254. View

Gandhi Z, Gurram P, Amgai B, Lekkala S, Lokhandwala A, Manne S . Artificial Intelligence and Lung Cancer: Impact on Improving Patient Outcomes. Cancers (Basel). 2023; 15(21). PMC: 10650618. DOI: 10.3390/cancers15215236. View

Knight S, Crosbie P, Balata H, Chudziak J, Hussell T, Dive C . Progress and prospects of early detection in lung cancer. Open Biol. 2017; 7(9). PMC: 5627048. DOI: 10.1098/rsob.170070. View

10.

Lambin P, Leijenaar R, Deist T, Peerlings J, de Jong E, van Timmeren J . Radiomics: the bridge between medical imaging and personalized medicine. Nat Rev Clin Oncol. 2017; 14(12):749-762. DOI: 10.1038/nrclinonc.2017.141. View

11.

Luo X, Song T, Wang G, Chen J, Chen Y, Li K . SCPM-Net: An anchor-free 3D lung nodule detection network using sphere representation and center points matching. Med Image Anal. 2021; 75:102287. DOI: 10.1016/j.media.2021.102287. View

12.

Choi E, Park D, Son G, Bak S, Eo T, Youn D . Weakly supervised deep learning for diagnosis of multiple vertebral compression fractures in CT. Eur Radiol. 2023; 34(6):3750-3760. DOI: 10.1007/s00330-023-10394-9. View

13.

Setio A, Traverso A, de Bel T, Berens M, Bogaard C, Cerello P . Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: The LUNA16 challenge. Med Image Anal. 2017; 42:1-13. DOI: 10.1016/j.media.2017.06.015. View

14.

Park D, Oh D, Lee M, Lee S, Shin K, Jun J . Importance of CT image normalization in radiomics analysis: prediction of 3-year recurrence-free survival in non-small cell lung cancer. Eur Radiol. 2022; 32(12):8716-8725. DOI: 10.1007/s00330-022-08869-2. View

15.

Christensen J, Prosper A, Wu C, Chung J, Lee E, Elicker B . ACR Lung-RADS v2022: Assessment Categories and Management Recommendations. J Am Coll Radiol. 2023; 21(3):473-488. DOI: 10.1016/j.jacr.2023.09.009. View

16.

Najjar R . Redefining Radiology: A Review of Artificial Intelligence Integration in Medical Imaging. Diagnostics (Basel). 2023; 13(17). PMC: 10487271. DOI: 10.3390/diagnostics13172760. View

17.

Nooreldeen R, Bach H . Current and Future Development in Lung Cancer Diagnosis. Int J Mol Sci. 2021; 22(16). PMC: 8395394. DOI: 10.3390/ijms22168661. View

18.

Al-Antari M, Al-Masni M, Choi M, Han S, Kim T . A fully integrated computer-aided diagnosis system for digital X-ray mammograms via deep learning detection, segmentation, and classification. Int J Med Inform. 2018; 117:44-54. DOI: 10.1016/j.ijmedinf.2018.06.003. View