Heatmap Analysis for Artificial Intelligence Explainability in Diabetic Retinopathy Detection: Illuminating the Rationale of Deep Learning Decisions

Overview

Journal Ann Transl Med

Publisher AME Publishing Company

Date 2024 Nov 7

PMID 39507460

Authors

Fernando Korn Malerbi

Luis Filipe Nakayama

Paulo Prado

Fernando Yamanaka

Gustavo Barreto Melo

Caio Vinicius Regatieri

Jose Augusto Stuchi

Affiliations

Soon will be listed here.

Abstract

Background: The opaqueness of artificial intelligence (AI) algorithms decision processes limit their application in healthcare. Our objective was to explore discrepancies in heatmaps originated from slightly different retinal images from the same eyes of individuals with diabetes, to gain insights into the deep learning (DL) decision process.

Methods: Pairs of retinal images from the same eyes of individuals with diabetes, composed of images obtained before and after pupil dilation, underwent automatic analysis by a convolutional neural network for the presence of diabetic retinopathy (DR), output being a score ranging from 0 to 1. Gradient-based Class Activation Maps (GradCam) allowed visualization of activated areas. Pairs of images with discordant DL scores or outputs within the pair were objectively compared to the concordant pairs, regarding the sum of activations of Class Activation Mapping (CAM), the number of activated areas, and DL score differences. Heatmaps of discordant pairs were also qualitatively assessed.

Results: Algorithmic performance for the detection of DR attained 89.8% sensitivity, 96.3% specificity and area under the receiver operating characteristic (ROC) curve of 0.95. Out of 210 comparable pairs of images, 20 eyes and 10 eyes were considered discordant according to DL score difference and regarding DL output, respectively. Comparison of concordant versus discordant groups showed statistically significant differences for all objective variables. Qualitative analysis pointed to subtle differences in image quality within discordant pairs.

Conclusions: The successfully established relationship among objective parameters extracted from heatmaps and DL output discrepancies reinforces the role of heatmaps for DL explainability, fostering acceptance of DL systems for clinical use.

References

Chang J, Lee J, Ha A, Han Y, Bak E, Choi S . Explaining the Rationale of Deep Learning Glaucoma Decisions with Adversarial Examples. Ophthalmology. 2020; 128(1):78-88. DOI: 10.1016/j.ophtha.2020.06.036. View

Korot E, Pontikos N, Liu X, Wagner S, Faes L, Huemer J . Predicting sex from retinal fundus photographs using automated deep learning. Sci Rep. 2021; 11(1):10286. PMC: 8119673. DOI: 10.1038/s41598-021-89743-x. View

Yousefi S, Pasquale L, Boland M . Artificial Intelligence and Glaucoma: Illuminating the Black Box. Ophthalmol Glaucoma. 2020; 3(5):311-313. DOI: 10.1016/j.ogla.2020.04.008. View

Amann J, Blasimme A, Vayena E, Frey D, Madai V . Explainability for artificial intelligence in healthcare: a multidisciplinary perspective. BMC Med Inform Decis Mak. 2020; 20(1):310. PMC: 7706019. DOI: 10.1186/s12911-020-01332-6. View

Quellec G, Al Hajj H, Lamard M, Conze P, Massin P, Cochener B . ExplAIn: Explanatory artificial intelligence for diabetic retinopathy diagnosis. Med Image Anal. 2021; 72:102118. DOI: 10.1016/j.media.2021.102118. View

Rajalakshmi R, Subashini R, Anjana R, Mohan V . Automated diabetic retinopathy detection in smartphone-based fundus photography using artificial intelligence. Eye (Lond). 2018; 32(6):1138-1144. PMC: 5997766. DOI: 10.1038/s41433-018-0064-9. View

Nunez do Rio J, Nderitu P, Bergeles C, Sivaprasad S, Tan G, Raman R . Evaluating a Deep Learning Diabetic Retinopathy Grading System Developed on Mydriatic Retinal Images When Applied to Non-Mydriatic Community Screening. J Clin Med. 2022; 11(3). PMC: 8836386. DOI: 10.3390/jcm11030614. View

Grzybowski A, Brona P, Lim G, Ruamviboonsuk P, Tan G, Abramoff M . Artificial intelligence for diabetic retinopathy screening: a review. Eye (Lond). 2019; 34(3):451-460. PMC: 7055592. DOI: 10.1038/s41433-019-0566-0. View

Poplin R, Varadarajan A, Blumer K, Liu Y, McConnell M, Corrado G . Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning. Nat Biomed Eng. 2019; 2(3):158-164. DOI: 10.1038/s41551-018-0195-0. View

10.

London A . Artificial Intelligence and Black-Box Medical Decisions: Accuracy versus Explainability. Hastings Cent Rep. 2019; 49(1):15-21. DOI: 10.1002/hast.973. View

11.

Esteva A, Robicquet A, Ramsundar B, Kuleshov V, DePristo M, Chou K . A guide to deep learning in healthcare. Nat Med. 2019; 25(1):24-29. DOI: 10.1038/s41591-018-0316-z. View

12.

Lim J, Regillo C, Sadda S, Ipp E, Bhaskaranand M, Ramachandra C . Artificial Intelligence Detection of Diabetic Retinopathy: Subgroup Comparison of the EyeArt System with Ophthalmologists' Dilated Examinations. Ophthalmol Sci. 2022; 3(1):100228. PMC: 9636573. DOI: 10.1016/j.xops.2022.100228. View

13.

Babic B, Gerke S, Evgeniou T, Cohen I . Beware explanations from AI in health care. Science. 2021; 373(6552):284-286. DOI: 10.1126/science.abg1834. View

14.

Malerbi F, Andrade R, Morales P, Stuchi J, Lencione D, de Paulo J . Diabetic Retinopathy Screening Using Artificial Intelligence and Handheld Smartphone-Based Retinal Camera. J Diabetes Sci Technol. 2021; 16(3):716-723. PMC: 9294565. DOI: 10.1177/1932296820985567. View

15.

LeCun Y, Bengio Y, Hinton G . Deep learning. Nature. 2015; 521(7553):436-44. DOI: 10.1038/nature14539. View

16.

Rajpurkar P, Chen E, Banerjee O, Topol E . AI in health and medicine. Nat Med. 2022; 28(1):31-38. DOI: 10.1038/s41591-021-01614-0. View

17.

Ruan S, Liu Y, Hu W, Jia H, Wang S, Song M . A new handheld fundus camera combined with visual artificial intelligence facilitates diabetic retinopathy screening. Int J Ophthalmol. 2022; 15(4):620-627. PMC: 8995718. DOI: 10.18240/ijo.2022.04.16. View

18.

Yu K, Beam A, Kohane I . Artificial intelligence in healthcare. Nat Biomed Eng. 2019; 2(10):719-731. DOI: 10.1038/s41551-018-0305-z. View

19.

Lupidi M, Danieli L, Fruttini D, Nicolai M, Lassandro N, Chhablani J . Artificial intelligence in diabetic retinopathy screening: clinical assessment using handheld fundus camera in a real-life setting. Acta Diabetol. 2023; 60(8):1083-1088. PMC: 10166040. DOI: 10.1007/s00592-023-02104-0. View

20.

Beam A, Drazen J, Kohane I, Leong T, Manrai A, Rubin E . Artificial Intelligence in Medicine. N Engl J Med. 2023; 388(13):1220-1221. DOI: 10.1056/NEJMe2206291. View