Prognostic Models Will Be Victims of Their Own Success, Unless…

Overview

Journal J Am Med Inform Assoc

Publisher Oxford University Press

Specialty Medical Informatics

Date 2019 Sep 11

PMID 31504588

Citations 37

Authors

Matthew C Lenert

Michael E Matheny

Colin G Walsh

Affiliations

Soon will be listed here.

Abstract

Predictive analytics have begun to change the workflows of healthcare by giving insight into our future health. Deploying prognostic models into clinical workflows should change behavior and motivate interventions that affect outcomes. As users respond to model predictions, downstream characteristics of the data, including the distribution of the outcome, may change. The ever-changing nature of healthcare necessitates maintenance of prognostic models to ensure their longevity. The more effective a model and intervention(s) are at improving outcomes, the faster a model will appear to degrade. Improving outcomes can disrupt the association between the model's predictors and the outcome. Model refitting may not always be the most effective response to these challenges. These problems will need to be mitigated by systematically incorporating interventions into prognostic models and by maintaining robust performance surveillance of models in clinical use. Holistically modeling the outcome and intervention(s) can lead to resilience to future compromises in performance.

Citing Articles

The early warning paradox.

Logan Ellis H, Palmer E, Teo J, Whyte M, Rockwood K, Ibrahim Z NPJ Digit Med. 2025; 8(1):81.

PMID: 39900787 PMC: 11790821. DOI: 10.1038/s41746-024-01408-x.

Proximity to Practice: The Role of Technology in the Next Era of Assessment.

Krumm A, Lai H, Marcotte K, Ark T, Yaneva V, Chahine S Perspect Med Educ. 2024; 13(1):646-653.

PMID: 39735827 PMC: 11673589. DOI: 10.5334/pme.1272.

Improving survival models in healthcare: a novel matching approach.

Bertsimas D, Ning C, Lonning P, Baba H, Endo I, Burkhart R Res Sq. 2024; .

PMID: 39711552 PMC: 11661418. DOI: 10.21203/rs.3.rs-5467577/v1.

Monitoring performance of clinical artificial intelligence in health care: a scoping review.

Andersen E, Birk-Korch J, Hansen R, Fly L, Rottger R, Arcani D JBI Evid Synth. 2024; 22(12):2423-2446.

PMID: 39658865 PMC: 11630661. DOI: 10.11124/JBIES-24-00042.

Development and assessment of a machine learning tool for predicting emergency admission in Scotland.

Liley J, Bohner G, Emerson S, Mateen B, Borland K, Carr D NPJ Digit Med. 2024; 7(1):277.

PMID: 39443624 PMC: 11499905. DOI: 10.1038/s41746-024-01250-1.

References

Fleming N, Becker E, Culler S, Cheng D, McCorkle R, da Graca B . The impact of electronic health records on workflow and financial measures in primary care practices. Health Serv Res. 2013; 49(1 Pt 2):405-20. PMC: 3925410. DOI: 10.1111/1475-6773.12133. View

Lorenzi N, Riley R . Managing change: an overview. J Am Med Inform Assoc. 2000; 7(2):116-24. PMC: 61464. DOI: 10.1136/jamia.2000.0070116. View

Moons K, Altman D, Vergouwe Y, Royston P . Prognosis and prognostic research: application and impact of prognostic models in clinical practice. BMJ. 2009; 338:b606. DOI: 10.1136/bmj.b606. View

Pencina M, DAgostino Sr R . Evaluating Discrimination of Risk Prediction Models: The C Statistic. JAMA. 2015; 314(10):1063-4. DOI: 10.1001/jama.2015.11082. View

Dressel J, Farid H . The accuracy, fairness, and limits of predicting recidivism. Sci Adv. 2018; 4(1):eaao5580. PMC: 5777393. DOI: 10.1126/sciadv.aao5580. View

Wiens J, Shenoy E . Machine Learning for Healthcare: On the Verge of a Major Shift in Healthcare Epidemiology. Clin Infect Dis. 2017; 66(1):149-153. PMC: 5850539. DOI: 10.1093/cid/cix731. View

Davis S, Lasko T, Chen G, Siew E, Matheny M . Calibration drift in regression and machine learning models for acute kidney injury. J Am Med Inform Assoc. 2017; 24(6):1052-1061. PMC: 6080675. DOI: 10.1093/jamia/ocx030. View

Bates D, Kuperman G, Wang S, Gandhi T, Kittler A, Volk L . Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc. 2003; 10(6):523-30. PMC: 264429. DOI: 10.1197/jamia.M1370. View

Miller R, Pople Jr H, Myers J . Internist-1, an experimental computer-based diagnostic consultant for general internal medicine. N Engl J Med. 1982; 307(8):468-76. DOI: 10.1056/NEJM198208193070803. View

10.

Steyerberg E, van der Ploeg T, Van Calster B . Risk prediction with machine learning and regression methods. Biom J. 2014; 56(4):601-6. DOI: 10.1002/bimj.201300297. View

11.

Amarasingham R, Patzer R, Huesch M, Nguyen N, Xie B . Implementing electronic health care predictive analytics: considerations and challenges. Health Aff (Millwood). 2014; 33(7):1148-54. DOI: 10.1377/hlthaff.2014.0352. View

12.

Collins S, Peacock W, Lindsell C, Clopton P, Diercks D, Hiestand B . S3 detection as a diagnostic and prognostic aid in emergency department patients with acute dyspnea. Ann Emerg Med. 2009; 53(6):748-57. DOI: 10.1016/j.annemergmed.2008.12.029. View

13.

Prasad V, Ioannidis J . Evidence-based de-implementation for contradicted, unproven, and aspiring healthcare practices. Implement Sci. 2014; 9:1. PMC: 3892018. DOI: 10.1186/1748-5908-9-1. View

14.

Wong H, Weiss S, Giuliano Jr J, Wainwright M, Cvijanovich N, Thomas N . Testing the prognostic accuracy of the updated pediatric sepsis biomarker risk model. PLoS One. 2014; 9(1):e86242. PMC: 3906040. DOI: 10.1371/journal.pone.0086242. View

15.

Albritton J, Belnap T, Savitz L . The Effect Of The Hospital Readmissions Reduction Program On Readmission And Observation Stay Rates For Heart Failure. Health Aff (Millwood). 2018; 37(10):1632-1639. DOI: 10.1377/hlthaff.2018.0064. View

16.

Moons K, Royston P, Vergouwe Y, Grobbee D, Altman D . Prognosis and prognostic research: what, why, and how?. BMJ. 2009; 338:b375. DOI: 10.1136/bmj.b375. View

17.

Miller R . Predictive models for primary caregivers: risky business?. Ann Intern Med. 1997; 127(7):565-7. DOI: 10.7326/0003-4819-127-7-199710010-00008. View

18.

Miller R, Schaffner K, Meisel A . Ethical and legal issues related to the use of computer programs in clinical medicine. Ann Intern Med. 1985; 102(4):529-37. DOI: 10.7326/0003-4819-102-4-529. View

19.

Hall M, Levant S, DeFrances C . Trends in inpatient hospital deaths: National Hospital Discharge Survey, 2000-2010. NCHS Data Brief. 2013; (118):1-8. View

20.

Parikh R, Kakad M, Bates D . Integrating Predictive Analytics Into High-Value Care: The Dawn of Precision Delivery. JAMA. 2016; 315(7):651-2. DOI: 10.1001/jama.2015.19417. View