Five Critical Quality Criteria for Artificial Intelligence-based Prediction Models

Overview

Journal Eur Heart J

Publisher Oxford University Press

Specialty Cardiology & Vascular Diseases

Date 2023 Oct 28

PMID 37897346

Authors

Florien S van Royen

Folkert W Asselbergs

Fernando Alfonso

Panos Vardas

Maarten van Smeden

Affiliations

Soon will be listed here.

Abstract

To raise the quality of clinical artificial intelligence (AI) prediction modelling studies in the cardiovascular health domain and thereby improve their impact and relevancy, the editors for digital health, innovation, and quality standards of the European Heart Journal propose five minimal quality criteria for AI-based prediction model development and validation studies: complete reporting, carefully defined intended use of the model, rigorous validation, large enough sample size, and openness of code and software.

Citing Articles

The Heart of Transformation: Exploring Artificial Intelligence in Cardiovascular Disease.

Chowdhury M, Rizk R, Chiu C, Zhang J, Scholl J, Bosch T Biomedicines. 2025; 13(2).

PMID: 40002840 PMC: 11852486. DOI: 10.3390/biomedicines13020427.

AI for IMPACTS Framework for Evaluating the Long-Term Real-World Impacts of AI-Powered Clinician Tools: Systematic Review and Narrative Synthesis.

Jacob C, Brasier N, Laurenzi E, Heuss S, Mougiakakou S, Coltekin A J Med Internet Res. 2025; 27:e67485.

PMID: 39909417 PMC: 11840377. DOI: 10.2196/67485.

Prediction of cardiovascular markers and diseases using retinal fundus images and deep learning: a systematic scoping review.

Li L, Isaksen A, Lebiecka-Johansen B, Funck K, Thambawita V, Byberg S Eur Heart J Digit Health. 2024; 5(6):660-669.

PMID: 39563905 PMC: 11570365. DOI: 10.1093/ehjdh/ztae068.

Prediction of incident atrial fibrillation using deep learning, clinical models, and polygenic scores.

Jabbour G, Nolin-Lapalme A, Tastet O, Corbin D, Jorda P, Sowa A Eur Heart J. 2024; 45(46):4920-4934.

PMID: 39217446 PMC: 11631091. DOI: 10.1093/eurheartj/ehae595.

Identification of avoidable patients at triage in a Paediatric Emergency Department: a decision support system using predictive analytics.

Viana J, Souza J, Rocha R, Santos A, Freitas A BMC Emerg Med. 2024; 24(1):149.

PMID: 39155373 PMC: 11331632. DOI: 10.1186/s12873-024-01029-3.

References

Wessler B, Nelson J, Park J, McGinnes H, Gulati G, Brazil R . External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature. Circ Cardiovasc Qual Outcomes. 2021; 14(8):e007858. PMC: 8366535. DOI: 10.1161/CIRCOUTCOMES.121.007858. View

Nakamura T, Sasano T . Artificial intelligence and cardiology: Current status and perspective. J Cardiol. 2021; 79(3):326-333. DOI: 10.1016/j.jjcc.2021.11.017. View

. SCORE2 risk prediction algorithms: new models to estimate 10-year risk of cardiovascular disease in Europe. Eur Heart J. 2021; 42(25):2439-2454. PMC: 8248998. DOI: 10.1093/eurheartj/ehab309. View

Moons K, Kengne A, Grobbee D, Royston P, Vergouwe Y, Altman D . Risk prediction models: II. External validation, model updating, and impact assessment. Heart. 2012; 98(9):691-8. DOI: 10.1136/heartjnl-2011-301247. View

Moons K, Altman D, Reitsma J, Ioannidis J, Macaskill P, Steyerberg E . Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015; 162(1):W1-73. DOI: 10.7326/M14-0698. View

Collins G, Reitsma J, Altman D, Moons K . Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement. Ann Intern Med. 2015; 162(1):55-63. DOI: 10.7326/M14-0697. View

Van Calster B, McLernon D, van Smeden M, Wynants L, Steyerberg E . Calibration: the Achilles heel of predictive analytics. BMC Med. 2019; 17(1):230. PMC: 6912996. DOI: 10.1186/s12916-019-1466-7. View

Van Calster B, Steyerberg E, Wynants L, van Smeden M . There is no such thing as a validated prediction model. BMC Med. 2023; 21(1):70. PMC: 9951847. DOI: 10.1186/s12916-023-02779-w. View

Nagendran M, Chen Y, Lovejoy C, Gordon A, Komorowski M, Harvey H . Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies. BMJ. 2020; 368:m689. PMC: 7190037. DOI: 10.1136/bmj.m689. View

10.

Gulati G, Upshaw J, Wessler B, Brazil R, Nelson J, van Klaveren D . Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models. Circ Cardiovasc Qual Outcomes. 2022; 15(4):e008487. PMC: 9015037. DOI: 10.1161/CIRCOUTCOMES.121.008487. View

11.

DAgostino Sr R, Vasan R, Pencina M, Wolf P, Cobain M, Massaro J . General cardiovascular risk profile for use in primary care: the Framingham Heart Study. Circulation. 2008; 117(6):743-53. DOI: 10.1161/CIRCULATIONAHA.107.699579. View

12.

Kotecha D, Asselbergs F, Achenbach S, Anker S, Atar D, Baigent C . CODE-EHR best practice framework for the use of structured electronic healthcare records in clinical research. Eur Heart J. 2022; 43(37):3578-3588. PMC: 9452067. DOI: 10.1093/eurheartj/ehac426. View

13.

Andaur Navarro C, Damen J, Takada T, Nijman S, Dhiman P, Ma J . Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review. BMJ. 2021; 375:n2281. PMC: 8527348. DOI: 10.1136/bmj.n2281. View

14.

Collins G, Moons K . Reporting of artificial intelligence prediction models. Lancet. 2019; 393(10181):1577-1579. DOI: 10.1016/S0140-6736(19)30037-6. View

15.

Collins G, Dhiman P, Andaur Navarro C, Ma J, Hooft L, Reitsma J . Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence. BMJ Open. 2021; 11(7):e048008. PMC: 8273461. DOI: 10.1136/bmjopen-2020-048008. View

16.

Riley R, Debray T, Collins G, Archer L, Ensor J, van Smeden M . Minimum sample size for external validation of a clinical prediction model with a binary outcome. Stat Med. 2021; 40(19):4230-4251. DOI: 10.1002/sim.9025. View

17.

Watson J, Hutyra C, Clancy S, Chandiramani A, Bedoya A, Ilangovan K . Overcoming barriers to the adoption and implementation of predictive modeling and machine learning in clinical care: what can we learn from US academic medical centers?. JAMIA Open. 2020; 3(2):167-172. PMC: 7382631. DOI: 10.1093/jamiaopen/ooz046. View

18.

van Smeden M, Moons K, de Groot J, Collins G, Altman D, Eijkemans M . Sample size for binary logistic prediction models: Beyond events per variable criteria. Stat Methods Med Res. 2018; 28(8):2455-2474. PMC: 6710621. DOI: 10.1177/0962280218784726. View

19.

van Smeden M, Heinze G, Van Calster B, Asselbergs F, Vardas P, Bruining N . Critical appraisal of artificial intelligence-based prediction models for cardiovascular disease. Eur Heart J. 2022; 43(31):2921-2930. PMC: 9443991. DOI: 10.1093/eurheartj/ehac238. View

20.

Alfonso F . Data Sharing. Eur Heart J. 2018; 38(18):1361-1363. DOI: 10.1093/eurheartj/ehx206. View