External Validation of Deep Learning Algorithms for Radiologic Diagnosis: A Systematic Review

Overview

Journal Radiol Artif Intell

Specialties Biomedical Engineering
Radiology

Date 2022 Jun 2

PMID 35652114

Authors

Alice C Yu

Bahram Mohajer

John Eng

Affiliations

Soon will be listed here.

Abstract

Purpose: To assess generalizability of published deep learning (DL) algorithms for radiologic diagnosis.

Materials And Methods: In this systematic review, the PubMed database was searched for peer-reviewed studies of DL algorithms for image-based radiologic diagnosis that included external validation, published from January 1, 2015, through April 1, 2021. Studies using nonimaging features or incorporating non-DL methods for feature extraction or classification were excluded. Two reviewers independently evaluated studies for inclusion, and any discrepancies were resolved by consensus. Internal and external performance measures and pertinent study characteristics were extracted, and relationships among these data were examined using nonparametric statistics.

Results: Eighty-three studies reporting 86 algorithms were included. The vast majority (70 of 86, 81%) reported at least some decrease in external performance compared with internal performance, with nearly half (42 of 86, 49%) reporting at least a modest decrease (≥0.05 on the unit scale) and nearly a quarter (21 of 86, 24%) reporting a substantial decrease (≥0.10 on the unit scale). No study characteristics were found to be associated with the difference between internal and external performance.

Conclusion: Among published external validation studies of DL algorithms for image-based radiologic diagnosis, the vast majority demonstrated diminished algorithm performance on the external dataset, with some reporting a substantial performance decrease. Meta-Analysis, Computer Applications-Detection/Diagnosis, Neural Networks, Computer Applications-General (Informatics), Epidemiology, Technology Assessment, Diagnosis, Informatics . © RSNA, 2022.

Citing Articles

Development and evaluation of a 3D ensemble framework for automatic diagnosis of early osteonecrosis of the femoral head based on MRI: a multicenter diagnostic study.

Yang M, Hsiang F, Li C, Chen X, Zhang C, Sun G Front Surg. 2025; 12:1555749.

PMID: 40026392 PMC: 11868283. DOI: 10.3389/fsurg.2025.1555749.

Deep Learning in Thoracic Oncology: Meta-Analytical Insights into Lung Nodule Early-Detection Technologies.

Wang T, Wang C, Hong J, Chao H, Chen Y, Wu Y Cancers (Basel). 2025; 17(4).

PMID: 40002216 PMC: 11853243. DOI: 10.3390/cancers17040621.

Sharing reliable information worldwide: healthcare strategies based on artificial intelligence need external validation. Position paper.

Pennestri F, Cabitza F, Picerno N, Banfi G BMC Med Inform Decis Mak. 2025; 25(1):56.

PMID: 39905337 PMC: 11796012. DOI: 10.1186/s12911-025-02883-2.

Use of AI in Cardiac CT and MRI: A Scientific Statement from the ESCR, EuSoMII, NASCI, SCCT, SCMR, SIIM, and RSNA.

Mastrodicasa D, van Assen M, Huisman M, Leiner T, Williamson E, Nicol E Radiology. 2025; 314(1):e240516.

PMID: 39873607 PMC: 11783164. DOI: 10.1148/radiol.240516.

External Validation of Deep Learning Models for Classifying Etiology of Retinal Hemorrhage Using Diverse Fundus Photography Datasets.

Khosravi P, Huck N, Shahraki K, Ghafari E, Azimi R, Kim S Bioengineering (Basel). 2025; 12(1).

PMID: 39851294 PMC: 11760437. DOI: 10.3390/bioengineering12010020.

References

Rudin C . Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead. Nat Mach Intell. 2022; 1(5):206-215. PMC: 9122117. DOI: 10.1038/s42256-019-0048-x. View

Mongan J, Moy L, Kahn Jr C . Checklist for Artificial Intelligence in Medical Imaging (CLAIM): A Guide for Authors and Reviewers. Radiol Artif Intell. 2021; 2(2):e200029. PMC: 8017414. DOI: 10.1148/ryai.2020200029. View

Xiao L, Li P, Sun F, Zhang Y, Xu C, Zhu H . Development and Validation of a Deep Learning-Based Model Using Computed Tomography Imaging for Predicting Disease Severity of Coronavirus Disease 2019. Front Bioeng Biotechnol. 2020; 8:898. PMC: 7411489. DOI: 10.3389/fbioe.2020.00898. View

Nael K, Gibson E, Yang C, Ceccaldi P, Yoo Y, Das J . Automated detection of critical findings in multi-parametric brain MRI using a system of 3D neural networks. Sci Rep. 2021; 11(1):6876. PMC: 7994311. DOI: 10.1038/s41598-021-86022-7. View

. AI diagnostics need attention. Nature. 2018; 555(7696):285. DOI: 10.1038/d41586-018-03067-x. View

Yao A, Cheng D, Pan I, Kitamura F . Deep Learning in Neuroradiology: A Systematic Review of Current Algorithms and Approaches for the New Wave of Imaging Technology. Radiol Artif Intell. 2021; 2(2):e190026. PMC: 8017426. DOI: 10.1148/ryai.2020190026. View

Suzuki K, Otsuka Y, Nomura Y, Kumamaru K, Kuwatsuru R, Aoki S . Development and Validation of a Modified Three-Dimensional U-Net Deep-Learning Model for Automated Detection of Lung Nodules on Chest CT Images From the Lung Image Database Consortium and Japanese Datasets. Acad Radiol. 2020; 29 Suppl 2:S11-S17. DOI: 10.1016/j.acra.2020.07.030. View

Gulshan V, Peng L, Coram M, Stumpe M, Wu D, Narayanaswamy A . Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. JAMA. 2016; 316(22):2402-2410. DOI: 10.1001/jama.2016.17216. View

Wu N, Phang J, Park J, Shen Y, Huang Z, Zorin M . Deep Neural Networks Improve Radiologists' Performance in Breast Cancer Screening. IEEE Trans Med Imaging. 2019; 39(4):1184-1194. PMC: 7427471. DOI: 10.1109/TMI.2019.2945514. View

10.

Schaffter T, Buist D, Lee C, Nikulin Y, Ribli D, Guan Y . Evaluation of Combined Artificial Intelligence and Radiologist Assessment to Interpret Screening Mammograms. JAMA Netw Open. 2020; 3(3):e200265. PMC: 7052735. DOI: 10.1001/jamanetworkopen.2020.0265. View

11.

Tang Y, Tang Y, Peng Y, Yan K, Bagheri M, Redd B . Automated abnormality classification of chest radiographs using deep convolutional neural networks. NPJ Digit Med. 2020; 3:70. PMC: 7224391. DOI: 10.1038/s41746-020-0273-z. View

12.

Lee J, Joo I, Kang T, Paik Y, Sinn D, Ha S . Deep learning with ultrasonography: automated classification of liver fibrosis using a deep convolutional neural network. Eur Radiol. 2019; 30(2):1264-1273. DOI: 10.1007/s00330-019-06407-1. View

13.

Bluthgen C, Becker A, Vittoria de Martini I, Meier A, Martini K, Frauenfelder T . Detection and localization of distal radius fractures: Deep learning system versus radiologists. Eur J Radiol. 2020; 126:108925. DOI: 10.1016/j.ejrad.2020.108925. View

14.

Dou Q, So T, Jiang M, Liu Q, Vardhanabhuti V, Kaissis G . Federated deep learning for detecting COVID-19 lung abnormalities in CT: a privacy-preserving multinational validation study. NPJ Digit Med. 2021; 4(1):60. PMC: 8007806. DOI: 10.1038/s41746-021-00431-6. View

15.

Kwon G, Ryu J, Oh J, Lim J, Kang B, Ahn C . Deep learning algorithms for detecting and visualising intussusception on plain abdominal radiography in children: a retrospective multicenter study. Sci Rep. 2020; 10(1):17582. PMC: 7567788. DOI: 10.1038/s41598-020-74653-1. View

16.

Rothwell P . Factors that can affect the external validity of randomised controlled trials. PLoS Clin Trials. 2006; 1(1):e9. PMC: 1488890. DOI: 10.1371/journal.pctr.0010009. View

17.

Kuo W, Hne C, Mukherjee P, Malik J, Yuh E . Expert-level detection of acute intracranial hemorrhage on head computed tomography using deep learning. Proc Natl Acad Sci U S A. 2019; 116(45):22737-22745. PMC: 6842581. DOI: 10.1073/pnas.1908021116. View

18.

Cole S, Stuart E . Generalizing evidence from randomized clinical trials to target populations: The ACTG 320 trial. Am J Epidemiol. 2010; 172(1):107-15. PMC: 2915476. DOI: 10.1093/aje/kwq084. View

19.

Bien N, Rajpurkar P, Ball R, Irvin J, Park A, Jones E . Deep-learning-assisted diagnosis for knee magnetic resonance imaging: Development and retrospective validation of MRNet. PLoS Med. 2018; 15(11):e1002699. PMC: 6258509. DOI: 10.1371/journal.pmed.1002699. View

20.

Blackmore C . The challenge of clinical radiology research. AJR Am J Roentgenol. 2001; 176(2):327-31. DOI: 10.2214/ajr.176.2.1760327. View