» Articles » PMID: 20807737

External Validity of Risk Models: Use of Benchmark Values to Disentangle a Case-mix Effect from Incorrect Coefficients

Overview
Journal Am J Epidemiol
Specialty Public Health
Date 2010 Sep 3
PMID 20807737
Citations 105
Authors
Affiliations
Soon will be listed here.
Abstract

Various performance measures related to calibration and discrimination are available for the assessment of risk models. When the validity of a risk model is assessed in a new population, estimates of the model's performance can be influenced in several ways. The regression coefficients can be incorrect, which indeed results in an invalid model. However, the distribution of patient characteristics (case mix) may also influence the performance of the model. Here the authors consider a number of typical situations that can be encountered in external validation studies. Theoretical relations between differences in development and validation samples and performance measures are studied by simulation. Benchmark values for the performance measures are proposed to disentangle a case-mix effect from incorrect regression coefficients, when interpreting the model's estimated performance in validation samples. The authors demonstrate the use of the benchmark values using data on traumatic brain injury obtained from the International Tirilazad Trial and the North American Tirilazad Trial (1991-1994).

Citing Articles

Genetics, primary care records and lifestyle factors for short-term dynamic risk prediction of colorectal cancer: prospective study of asymptomatic and symptomatic UK Biobank participants.

Ip S, Harrison H, Usher-Smith J, Barclay M, Tyrer J, Dennis J BMJ Oncol. 2025; 4(1):e000336.

PMID: 40046831 PMC: 11880779. DOI: 10.1136/bmjonc-2024-000336.


Pitfalls of single-study external validation illustrated with a model predicting functional outcome after aneurysmal subarachnoid hemorrhage.

de Winkel J, Maas C, Roozenbeek B, van Klaveren D, Lingsma H BMC Med Res Methodol. 2024; 24(1):176.

PMID: 39118007 PMC: 11308226. DOI: 10.1186/s12874-024-02280-9.


A constrained maximum likelihood approach to developing well-calibrated models for predicting binary outcomes.

Cao Y, Ma W, Zhao G, McCarthy A, Chen J Lifetime Data Anal. 2024; 30(3):624-648.

PMID: 38717617 PMC: 11634939. DOI: 10.1007/s10985-024-09628-9.


External validation and update of the International Medical Prevention Registry on Venous Thromboembolism bleeding risk score for predicting bleeding in acutely ill hospitalized medical patients: a retrospective single-center cohort study in Japan.

Arakaki D, Iwata M, Terasawa T Thromb J. 2024; 22(1):31.

PMID: 38549086 PMC: 10976666. DOI: 10.1186/s12959-024-00603-w.


Evaluation of clinical prediction models (part 2): how to undertake an external validation study.

Riley R, Archer L, Snell K, Ensor J, Dhiman P, Martin G BMJ. 2024; 384:e074820.

PMID: 38224968 PMC: 10788734. DOI: 10.1136/bmj-2023-074820.


References
1.
Justice A, Covinsky K, Berlin J . Assessing the generalizability of prognostic information. Ann Intern Med. 1999; 130(6):515-24. DOI: 10.7326/0003-4819-130-6-199903160-00016. View

2.
Gail M, Brinton L, Byar D, Corle D, Green S, Schairer C . Projecting individualized probabilities of developing breast cancer for white females who are being examined annually. J Natl Cancer Inst. 1989; 81(24):1879-86. DOI: 10.1093/jnci/81.24.1879. View

3.
van Houwelingen H, Thorogood J . Construction, validation and updating of a prognostic model for kidney graft survival. Stat Med. 1995; 14(18):1999-2008. DOI: 10.1002/sim.4780141806. View

4.
Mushkudiani N, Hukkelhoven C, Hernandez A, Murray G, Choi S, Maas A . A systematic review finds methodological improvements necessary for prognostic models in determining traumatic brain injury outcomes. J Clin Epidemiol. 2008; 61(4):331-43. DOI: 10.1016/j.jclinepi.2007.06.011. View

5.
Steyerberg E, Mushkudiani N, Perel P, Butcher I, Lu J, McHugh G . Predicting outcome after traumatic brain injury: development and international validation of prognostic scores based on admission characteristics. PLoS Med. 2008; 5(8):e165. PMC: 2494563. DOI: 10.1371/journal.pmed.0050165. View