Evaluation of Clinical Prediction Models (part 1): from Development to External Validation

Overview

Journal BMJ

Specialty General Medicine

Date 2024 Jan 8

PMID 38191193

Authors

Gary S Collins

Paula Dhiman

Jie Ma

Michael M Schlussel

Lucinda Archer

Ben Van Calster

Frank E Harrell Jr

Glen P Martin

Karel G M Moons

Maarten van Smeden

Matthew Sperrin

Garrett S Bullock

Richard D Riley

Affiliations

Soon will be listed here.

Citing Articles

Comparing conventional and Bayesian workflows for clinical outcome prediction modelling with an exemplar cohort study of severe COVID-19 infection incorporating clinical biomarker test results.

Sullivan B, Barker E, MacGregor L, Gorman L, Williams P, Bhamber R BMC Med Inform Decis Mak. 2025; 25(1):123.

PMID: 40065374 PMC: 11892292. DOI: 10.1186/s12911-025-02955-3.

The development and validation of a prediction model for post-AKI outcomes of pediatric inpatients.

Zhang C, Liu X, Yan R, Nie X, Peng Y, Zhou N Clin Kidney J. 2025; 18(2):sfaf007.

PMID: 39991652 PMC: 11843026. DOI: 10.1093/ckj/sfaf007.

Effect of an unsupervised multidomain intervention integrating education, exercises, psychological techniques and machine learning feedback, on injury risk reduction in athletics (track and field): protocol of a randomised controlled trial....

Iatropoulos S, Dandrieux P, Blanco D, Ruffault A, Gignoux E, Mosser C BMJ Open Sport Exerc Med. 2025; 11(1):e002501.

PMID: 39990118 PMC: 11843022. DOI: 10.1136/bmjsem-2025-002501.

The urgency of centering safety-net organizations in AI governance.

Nong P, Maurer E, Dwivedi R NPJ Digit Med. 2025; 8(1):117.

PMID: 39984650 PMC: 11845669. DOI: 10.1038/s41746-025-01479-4.

Validation of 30-Day Pediatric Hospital Readmission Risk Prediction Models.

Carroll A, Hall M, Harris M, Carroll M, Auger K, Davis M JAMA Netw Open. 2025; 8(2):e2459684.

PMID: 39946127 PMC: 11826366. DOI: 10.1001/jamanetworkopen.2024.59684.

References

Wessler B, Nelson J, Park J, McGinnes H, Gulati G, Brazil R . External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature. Circ Cardiovasc Qual Outcomes. 2021; 14(8):e007858. PMC: 8366535. DOI: 10.1161/CIRCOUTCOMES.121.007858. View

Justice A, Covinsky K, Berlin J . Assessing the generalizability of prognostic information. Ann Intern Med. 1999; 130(6):515-24. DOI: 10.7326/0003-4819-130-6-199903160-00016. View

Christodoulou E, Ma J, Collins G, Steyerberg E, Verbakel J, Van Calster B . A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. J Clin Epidemiol. 2019; 110:12-22. DOI: 10.1016/j.jclinepi.2019.02.004. View

Riley R, Snell K, Ensor J, Burke D, Harrell Jr F, Moons K . Minimum sample size for developing a multivariable prediction model: PART II - binary and time-to-event outcomes. Stat Med. 2018; 38(7):1276-1296. PMC: 6519266. DOI: 10.1002/sim.7992. View

Debray T, Collins G, Riley R, Snell K, Van Calster B, Reitsma J . Transparent reporting of multivariable prediction models developed or validated using clustered data (TRIPOD-Cluster): explanation and elaboration. BMJ. 2023; 380:e071058. PMC: 9903176. DOI: 10.1136/bmj-2022-071058. View

Takada T, Nijman S, Denaxas S, Snell K, Uijl A, Nguyen T . Internal-external cross-validation helped to evaluate the generalizability of prediction models in large clustered datasets. J Clin Epidemiol. 2021; 137:83-91. DOI: 10.1016/j.jclinepi.2021.03.025. View

Nevin L . Advancing the beneficial use of machine learning in health care and medicine: Toward a community understanding. PLoS Med. 2018; 15(11):e1002708. PMC: 6267950. DOI: 10.1371/journal.pmed.1002708. View

Gupta R, Harrison E, Ho A, Docherty A, Knight S, van Smeden M . Development and validation of the ISARIC 4C Deterioration model for adults hospitalised with COVID-19: a prospective cohort study. Lancet Respir Med. 2021; 9(4):349-359. PMC: 7832571. DOI: 10.1016/S2213-2600(20)30559-2. View

Debray T, Moons K, Ahmed I, Koffijberg H, Riley R . A framework for developing, implementing, and evaluating clinical prediction models in an individual participant data meta-analysis. Stat Med. 2013; 32(18):3158-80. DOI: 10.1002/sim.5732. View

10.

Shakur H, Roberts I, Bautista R, Caballero J, Coats T, Dewan Y . Effects of tranexamic acid on death, vascular occlusive events, and blood transfusion in trauma patients with significant haemorrhage (CRASH-2): a randomised, placebo-controlled trial. Lancet. 2010; 376(9734):23-32. DOI: 10.1016/S0140-6736(10)60835-5. View

11.

Steyerberg E . Validation in prediction research: the waste by data splitting. J Clin Epidemiol. 2018; 103:131-133. DOI: 10.1016/j.jclinepi.2018.07.010. View

12.

Collins G, Reitsma J, Altman D, Moons K . Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ. 2015; 350:g7594. DOI: 10.1136/bmj.g7594. View

13.

Riley R, Ensor J, Snell K, Debray T, Altman D, Moons K . External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges. BMJ. 2016; 353:i3140. PMC: 4916924. DOI: 10.1136/bmj.i3140. View

14.

van Smeden M, Moons K, de Groot J, Collins G, Altman D, Eijkemans M . Sample size for binary logistic prediction models: Beyond events per variable criteria. Stat Methods Med Res. 2018; 28(8):2455-2474. PMC: 6710621. DOI: 10.1177/0962280218784726. View

15.

Hickey G, Grant S, Murphy G, Bhabra M, Pagano D, McAllister K . Dynamic trends in cardiac surgery: why the logistic EuroSCORE is no longer suitable for contemporary cardiac surgery and implications for future risk models. Eur J Cardiothorac Surg. 2012; 43(6):1146-52. PMC: 3655624. DOI: 10.1093/ejcts/ezs584. View

16.

Moons K, Altman D, Vergouwe Y, Royston P . Prognosis and prognostic research: application and impact of prognostic models in clinical practice. BMJ. 2009; 338:b606. DOI: 10.1136/bmj.b606. View

17.

Royston P, Parmar M, Sylvester R . Construction and validation of a prognostic model across several studies, with an application in superficial bladder cancer. Stat Med. 2004; 23(6):907-26. DOI: 10.1002/sim.1691. View

18.

van Smeden M, de Groot J, Moons K, Collins G, Altman D, Eijkemans M . No rationale for 1 variable per 10 events criterion for binary logistic regression analysis. BMC Med Res Methodol. 2016; 16(1):163. PMC: 5122171. DOI: 10.1186/s12874-016-0267-3. View

19.

Damen J, Hooft L, Schuit E, Debray T, Collins G, Tzoulaki I . Prediction models for cardiovascular disease risk in the general population: systematic review. BMJ. 2016; 353:i2416. PMC: 4868251. DOI: 10.1136/bmj.i2416. View

20.

Davis S, Lasko T, Chen G, Siew E, Matheny M . Calibration drift in regression and machine learning models for acute kidney injury. J Am Med Inform Assoc. 2017; 24(6):1052-1061. PMC: 6080675. DOI: 10.1093/jamia/ocx030. View