Guidelines on Clinical Research Evaluation of Artificial Intelligence in Ophthalmology (2023)

Overview

Journal Int J Ophthalmol

Specialty Ophthalmology

Date 2023 Sep 19

PMID 37724285

Authors

Wei-Hua Yang

Yi Shao

Yan-Wu Xu

Affiliations

Soon will be listed here.

Abstract

With the upsurge of artificial intelligence (AI) technology in the medical field, its application in ophthalmology has become a cutting-edge research field. Notably, machine learning techniques have shown remarkable achievements in diagnosing, intervening, and predicting ophthalmic diseases. To meet the requirements of clinical research and fit the actual progress of clinical diagnosis and treatment of ophthalmic AI, the Ophthalmic Imaging and Intelligent Medicine Branch and the Intelligent Medicine Committee of Chinese Medicine Education Association organized experts to integrate recent evaluation reports of clinical AI research at home and abroad and formed a guideline on clinical research evaluation of AI in ophthalmology after several rounds of discussion and modification. The main content includes the background and method of developing this guideline, an introduction to international guidelines on the clinical research evaluation of AI, and the evaluation methods of clinical ophthalmic AI models. This guideline introduces general evaluation methods of clinical ophthalmic AI research, evaluation methods of clinical ophthalmic AI models, and commonly-used indices and formulae for clinical ophthalmic AI model evaluation in detail, and amply elaborates the evaluation methods of clinical ophthalmic AI trials. This guideline aims to provide guidance and norms for clinical researchers of ophthalmic AI, promote the development of regularization and standardization, and further improve the overall level of clinical ophthalmic AI research evaluations.

Citing Articles

Pathways to chronic disease detection and prediction: Mapping the potential of machine learning to the pathophysiological processes while navigating ethical challenges.

Afrifa-Yamoah E, Adua E, Peprah-Yamoah E, Anto E, Opoku-Yamoah V, Acheampong E Chronic Dis Transl Med. 2025; 11(1):1-21.

PMID: 40051825 PMC: 11880127. DOI: 10.1002/cdt3.137.

An Intelligent Grading Model for Myopic Maculopathy Based on Long-Tailed Learning.

Zheng B, Wang C, Zhang M, Zhu S, Wu M, Wu T Transl Vis Sci Technol. 2025; 14(3):4.

PMID: 40048172 PMC: 11895850. DOI: 10.1167/tvst.14.3.4.

Comparative analysis of retinal vascular structural parameters in populations with different glucose metabolism status based on color fundus photography and artificial intelligence.

Chen N, Zhu Z, Gong D, Xu X, Hu X, Yang W Front Cell Dev Biol. 2025; 13:1550176.

PMID: 39963156 PMC: 11830720. DOI: 10.3389/fcell.2025.1550176.

Exploring the effect of gestational diabetes mellitus on retinal vascular morphology by PKSEA-Net.

Jiang L, Ji Y, Liu M, Fang R, Zhu Z, Zhang M Front Cell Dev Biol. 2025; 12():1532939.

PMID: 39845084 PMC: 11750853. DOI: 10.3389/fcell.2024.1532939.

A multi-modal multi-branch framework for retinal vessel segmentation using ultra-widefield fundus photographs.

Xie Q, Li X, Li Y, Lu J, Ma S, Zhao Y Front Cell Dev Biol. 2025; 12():1532228.

PMID: 39845080 PMC: 11751237. DOI: 10.3389/fcell.2024.1532228.

References

Weiskopf N, Weng C . Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research. J Am Med Inform Assoc. 2012; 20(1):144-51. PMC: 3555312. DOI: 10.1136/amiajnl-2011-000681. View

Ruan S, Liu Y, Hu W, Jia H, Wang S, Song M . A new handheld fundus camera combined with visual artificial intelligence facilitates diabetic retinopathy screening. Int J Ophthalmol. 2022; 15(4):620-627. PMC: 8995718. DOI: 10.18240/ijo.2022.04.16. View

Moons K, Altman D, Vergouwe Y, Royston P . Prognosis and prognostic research: application and impact of prognostic models in clinical practice. BMJ. 2009; 338:b606. DOI: 10.1136/bmj.b606. View

Alba A, Agoritsas T, Walsh M, Hanna S, Iorio A, Devereaux P . Discrimination and Calibration of Clinical Prediction Models: Users' Guides to the Medical Literature. JAMA. 2017; 318(14):1377-1384. DOI: 10.1001/jama.2017.12126. View

Nattino G, Pennell M, Lemeshow S . Assessing the goodness of fit of logistic regression models in large samples: A modification of the Hosmer-Lemeshow test. Biometrics. 2020; 76(2):549-560. DOI: 10.1111/biom.13249. View

Yang W, Zheng B, Wu M, Zhu S, Fei F, Weng M . An Evaluation System of Fundus Photograph-Based Intelligent Diagnostic Technology for Diabetic Retinopathy and Applicability for Research. Diabetes Ther. 2019; 10(5):1811-1822. PMC: 6778552. DOI: 10.1007/s13300-019-0652-0. View

Salmi L, Saillour-Glenisson F, Alla F, Boussat B . [Evaluation and research on public health interventions]. Rev Epidemiol Sante Publique. 2023; 71(2):101836. DOI: 10.1016/j.respe.2023.101836. View

Skivington K, Matthews L, Simpson S, Craig P, Baird J, Blazeby J . A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance. BMJ. 2021; 374:n2061. PMC: 8482308. DOI: 10.1136/bmj.n2061. View

Savoy M . IDx-DR for Diabetic Retinopathy Screening. Am Fam Physician. 2020; 101(5):307-308. View

10.

Liu X, Cruz Rivera S, Moher D, Calvert M, Denniston A . Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. Lancet Digit Health. 2020; 2(10):e537-e548. PMC: 8183333. DOI: 10.1016/S2589-7500(20)30218-1. View

11.

Jordan M, Mitchell T . Machine learning: Trends, perspectives, and prospects. Science. 2015; 349(6245):255-60. DOI: 10.1126/science.aaa8415. View

12.

Cruz Rivera S, Liu X, Chan A, Denniston A, Calvert M . Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension. Lancet Digit Health. 2020; 2(10):e549-e560. PMC: 8212701. DOI: 10.1016/S2589-7500(20)30219-3. View

13.

Sounderajah V, Ashrafian H, Golub R, Shetty S, De Fauw J, Hooft L . Developing a reporting guideline for artificial intelligence-centred diagnostic test accuracy studies: the STARD-AI protocol. BMJ Open. 2021; 11(6):e047709. PMC: 8240576. DOI: 10.1136/bmjopen-2020-047709. View

14.

Han R, Cheng G, Zhang B, Yang J, Yuan M, Yang D . Validating automated eye disease screening AI algorithm in community and in-hospital scenarios. Front Public Health. 2022; 10:944967. PMC: 9354491. DOI: 10.3389/fpubh.2022.944967. View

15.

He J, Cao T, Xu F, Wang S, Tao H, Wu T . Artificial intelligence-based screening for diabetic retinopathy at community hospital. Eye (Lond). 2019; 34(3):572-576. PMC: 7042314. DOI: 10.1038/s41433-019-0562-4. View

16.

Fernandez Escamez C, Martin Giral E, Perucho Martinez S, Toledano Fernandez N . High interpretable machine learning classifier for early glaucoma diagnosis. Int J Ophthalmol. 2021; 14(3):393-398. PMC: 7930548. DOI: 10.18240/ijo.2021.03.10. View

17.

LeCun Y, Bengio Y, Hinton G . Deep learning. Nature. 2015; 521(7553):436-44. DOI: 10.1038/nature14539. View

18.

Collins G, Dhiman P, Andaur Navarro C, Ma J, Hooft L, Reitsma J . Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence. BMJ Open. 2021; 11(7):e048008. PMC: 8273461. DOI: 10.1136/bmjopen-2020-048008. View

19.

Moore G, Audrey S, Barker M, Bond L, Bonell C, Hardeman W . Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015; 350:h1258. PMC: 4366184. DOI: 10.1136/bmj.h1258. View

20.

Esteva A, Chou K, Yeung S, Naik N, Madani A, Mottaghi A . Deep learning-enabled medical computer vision. NPJ Digit Med. 2021; 4(1):5. PMC: 7794558. DOI: 10.1038/s41746-020-00376-2. View