Evaluation of Clinical Prediction Models (part 2): How to Undertake an External Validation Study

Overview

Journal BMJ

Specialty General Medicine

Date 2024 Jan 15

PMID 38224968

Authors

Richard D Riley

Lucinda Archer

Kym I E Snell

Joie Ensor

Paula Dhiman

Glen P Martin

Laura J Bonnett

Gary S Collins

Affiliations

Soon will be listed here.

Citing Articles

Comparing conventional and Bayesian workflows for clinical outcome prediction modelling with an exemplar cohort study of severe COVID-19 infection incorporating clinical biomarker test results.

Sullivan B, Barker E, MacGregor L, Gorman L, Williams P, Bhamber R BMC Med Inform Decis Mak. 2025; 25(1):123.

PMID: 40065374 PMC: 11892292. DOI: 10.1186/s12911-025-02955-3.

The Heart of Transformation: Exploring Artificial Intelligence in Cardiovascular Disease.

Chowdhury M, Rizk R, Chiu C, Zhang J, Scholl J, Bosch T Biomedicines. 2025; 13(2).

PMID: 40002840 PMC: 11852486. DOI: 10.3390/biomedicines13020427.

Development and validation of machine learning models based on stacked generalization to predict psychosocial maladjustment in patients with acute myocardial infarction.

Wang Y, Li X, Zhou X, Ke Q, Ma H, Li Z BMC Psychiatry. 2025; 25(1):152.

PMID: 39972470 PMC: 11841291. DOI: 10.1186/s12888-025-06549-1.

Expected Value of Sample Information Calculations for Risk Prediction Model Validation.

Sadatsafavi M, Vickers A, Lee T, Gustafson P, Wynants L Med Decis Making. 2025; :272989X251314010.

PMID: 39963746 PMC: 11894915. DOI: 10.1177/0272989X251314010.

Validation of a clinical prediction model for falls in community-dwelling older adults with COPD: A preliminary analysis.

Nguyen K, Ellerton C, Wald J, Raghavan N, Macedo L, Brooks D Chron Respir Dis. 2025; 22:14799731251321494.

PMID: 39957244 PMC: 11831686. DOI: 10.1177/14799731251321494.

References

Gerds T, Schumacher M . Consistent estimation of the expected Brier score in general survival models with right-censored event times. Biom J. 2007; 48(6):1029-40. DOI: 10.1002/bimj.200610301. View

Royston P, Altman D . External validation of a Cox prognostic model: principles and methods. BMC Med Res Methodol. 2013; 13:33. PMC: 3667097. DOI: 10.1186/1471-2288-13-33. View

Riley R, Debray T, Collins G, Archer L, Ensor J, van Smeden M . Minimum sample size for external validation of a clinical prediction model with a binary outcome. Stat Med. 2021; 40(19):4230-4251. DOI: 10.1002/sim.9025. View

Steyerberg E, Vergouwe Y . Towards better clinical prediction models: seven steps for development and an ABCD for validation. Eur Heart J. 2014; 35(29):1925-31. PMC: 4155437. DOI: 10.1093/eurheartj/ehu207. View

Russell Localio A, Goodman S . Beyond the usual prediction accuracy metrics: reporting results for clinical decision making. Ann Intern Med. 2012; 157(4):294-5. DOI: 10.7326/0003-4819-157-4-201208210-00014. View

Collins G, Reitsma J, Altman D, Moons K . Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement. Ann Intern Med. 2015; 162(1):55-63. DOI: 10.7326/M14-0697. View

Collins G, Ogundimu E, Altman D . Sample size considerations for the external validation of a multivariable prognostic model: a resampling study. Stat Med. 2015; 35(2):214-26. PMC: 4738418. DOI: 10.1002/sim.6787. View

Riley R, Collins G, Ensor J, Archer L, Booth S, Mozumder S . Minimum sample size calculations for external validation of a clinical prediction model with a time-to-event outcome. Stat Med. 2021; 41(7):1280-1295. DOI: 10.1002/sim.9275. View

Riley R, Snell K, Archer L, Ensor J, Debray T, Van Calster B . Evaluation of clinical prediction models (part 3): calculating the sample size required for an external validation study. BMJ. 2024; 384:e074821. PMC: 11778934. DOI: 10.1136/bmj-2023-074821. View

10.

Vergouwe Y, Moons K, Steyerberg E . External validity of risk models: Use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol. 2010; 172(8):971-80. PMC: 2984249. DOI: 10.1093/aje/kwq223. View

11.

Royston P, Sauerbrei W . A new measure of prognostic separation in survival data. Stat Med. 2004; 23(5):723-48. DOI: 10.1002/sim.1621. View

12.

Justice A, Covinsky K, Berlin J . Assessing the generalizability of prognostic information. Ann Intern Med. 1999; 130(6):515-24. DOI: 10.7326/0003-4819-130-6-199903160-00016. View

13.

Wolff R, Moons K, Riley R, Whiting P, Westwood M, Collins G . PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies. Ann Intern Med. 2019; 170(1):51-58. DOI: 10.7326/M18-1376. View

14.

Vickers A, Cronin A, Elkin E, Gonen M . Extensions to decision curve analysis, a novel method for evaluating diagnostic tests, prediction models and molecular markers. BMC Med Inform Decis Mak. 2008; 8:53. PMC: 2611975. DOI: 10.1186/1472-6947-8-53. View

15.

Collins G, Dhiman P, Ma J, Schlussel M, Archer L, Van Calster B . Evaluation of clinical prediction models (part 1): from development to external validation. BMJ. 2024; 384:e074819. PMC: 10772854. DOI: 10.1136/bmj-2023-074819. View

16.

Binuya M, Engelhardt E, Schats W, Schmidt M, Steyerberg E . Methodological guidance for the evaluation and updating of clinical prediction models: a systematic review. BMC Med Res Methodol. 2022; 22(1):316. PMC: 9742671. DOI: 10.1186/s12874-022-01801-8. View

17.

Riley R, Ensor J, Snell K, Debray T, Altman D, Moons K . External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges. BMJ. 2016; 353:i3140. PMC: 4916924. DOI: 10.1136/bmj.i3140. View

18.

Van Hoorde K, Van Huffel S, Timmerman D, Bourne T, Van Calster B . A spline-based tool to assess and visualize the calibration of multiclass risk predictions. J Biomed Inform. 2015; 54:283-93. DOI: 10.1016/j.jbi.2014.12.016. View

19.

Van Calster B, Wynants L, Collins G, Verbakel J, Steyerberg E . ROC curves for clinical prediction models part 3. The ROC plot: a picture that needs a 1000 words. J Clin Epidemiol. 2020; 126:220-223. DOI: 10.1016/j.jclinepi.2020.05.037. View

20.

Moons K, Altman D, Vergouwe Y, Royston P . Prognosis and prognostic research: application and impact of prognostic models in clinical practice. BMJ. 2009; 338:b606. DOI: 10.1136/bmj.b606. View