Calibration Drift in Regression and Machine Learning Models for Acute Kidney Injury

Overview

Journal J Am Med Inform Assoc

Publisher Oxford University Press

Specialty Medical Informatics

Date 2017 Apr 6

PMID 28379439

Citations 101

Authors

Sharon E Davis

Thomas A Lasko

Guanhua Chen

Edward D Siew

Michael E Matheny

Affiliations

Soon will be listed here.

Abstract

Objective: Predictive analytics create opportunities to incorporate personalized risk estimates into clinical decision support. Models must be well calibrated to support decision-making, yet calibration deteriorates over time. This study explored the influence of modeling methods on performance drift and connected observed drift with data shifts in the patient population.

Materials And Methods: Using 2003 admissions to Department of Veterans Affairs hospitals nationwide, we developed 7 parallel models for hospital-acquired acute kidney injury using common regression and machine learning methods, validating each over 9 subsequent years.

Results: Discrimination was maintained for all models. Calibration declined as all models increasingly overpredicted risk. However, the random forest and neural network models maintained calibration across ranges of probability, capturing more admissions than did the regression models. The magnitude of overprediction increased over time for the regression models while remaining stable and small for the machine learning models. Changes in the rate of acute kidney injury were strongly linked to increasing overprediction, while changes in predictor-outcome associations corresponded with diverging patterns of calibration drift across methods.

Conclusions: Efficient and effective updating protocols will be essential for maintaining accuracy of, user confidence in, and safety of personalized risk predictions to support decision-making. Model updating protocols should be tailored to account for variations in calibration drift across methods and respond to periods of rapid performance drift rather than be limited to regularly scheduled annual or biannual intervals.

Citing Articles

An algorithm to assess importance of predictors in systematic reviews of prediction models: a case study with simulations.

Yan R, Wang C, Zhang C, Liu X, Zhang D, Peng X BMC Med Res Methodol. 2025; 25(1):38.

PMID: 39953476 PMC: 11827416. DOI: 10.1186/s12874-025-02492-7.

Study protocol: Comparison of different risk prediction modelling approaches for COVID-19 related death using the OpenSAFELY platform.

Williamson E, Tazare J, Bhaskaran K, Walker A, McDonald H, Tomlinson L Wellcome Open Res. 2025; 5:243.

PMID: 39931522 PMC: 11809169. DOI: 10.12688/wellcomeopenres.16353.2.

Discrimination and calibration performances of non-laboratory-based and laboratory-based cardiovascular risk predictions: a systematic review.

Alemu Y, Alemu S, Bagheri N, Wangdi K, Chateau D Open Heart. 2025; 12(1).

PMID: 39929598 PMC: 11815431. DOI: 10.1136/openhrt-2024-003147.

Consecutive prediction of adverse maternal outcomes of preeclampsia, using the PIERS-ML and fullPIERS models: A multicountry prospective observational study.

Yang G, Montgomery-Csoban T, Ganzevoort W, Gordijn S, Kavanagh K, Murray P PLoS Med. 2025; 22(2):e1004509.

PMID: 39903757 PMC: 11793762. DOI: 10.1371/journal.pmed.1004509.

Medical Imaging Data Strategies for Catalyzing AI Medical Device Innovation.

Samala R, Gallas B, Zamzmi G, Juluru K, Khan A, Bahr C J Imaging Inform Med. 2025; .

PMID: 39881094 DOI: 10.1007/s10278-024-01374-6.

References

Slankamenac K, Beck-Schimmer B, Breitenstein S, Puhan M, Clavien P . Novel prediction score including pre- and intraoperative parameters best predicts acute kidney injury after liver surgery. World J Surg. 2013; 37(11):2618-28. DOI: 10.1007/s00268-013-2159-6. View

Park M, Shim H, Kim W, Kim H, Kim D, Lee S . Clinical Risk Scoring Models for Prediction of Acute Kidney Injury after Living Donor Liver Transplantation: A Retrospective Observational Study. PLoS One. 2015; 10(8):e0136230. PMC: 4547769. DOI: 10.1371/journal.pone.0136230. View

Malley J, Kruppa J, Dasgupta A, Malley K, Ziegler A . Probability machines: consistent probability estimation using nonparametric learning machines. Methods Inf Med. 2011; 51(1):74-81. PMC: 3250568. DOI: 10.3414/ME00-01-0052. View

Matheny M, Ohno-Machado L, Resnic F . Discrimination and calibration of mortality risk prediction models in interventional cardiology. J Biomed Inform. 2005; 38(5):367-75. DOI: 10.1016/j.jbi.2005.02.007. View

Mallett S, Royston P, Waters R, Dutton S, Altman D . Reporting performance of prognostic models in cancer: a review. BMC Med. 2010; 8:21. PMC: 2857810. DOI: 10.1186/1741-7015-8-21. View

Debray T, Vergouwe Y, Koffijberg H, Nieboer D, Steyerberg E, Moons K . A new framework to enhance the interpretation of external validation studies of clinical prediction models. J Clin Epidemiol. 2014; 68(3):279-89. DOI: 10.1016/j.jclinepi.2014.06.018. View

Hanley J, McNeil B . The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982; 143(1):29-36. DOI: 10.1148/radiology.143.1.7063747. View

Madan P, Elayda M, Lee V, Wilson J . Risk-prediction models for mortality after coronary artery bypass surgery: application to individual patients. Int J Cardiol. 2010; 149(2):227-231. DOI: 10.1016/j.ijcard.2010.02.005. View

Amarasingham R, Audet A, Bates D, Cohen I, Entwistle M, Escobar G . Consensus Statement on Electronic Health Predictive Analytics: A Guiding Framework to Address Challenges. EGEMS (Wash DC). 2016; 4(1):1163. PMC: 4837887. DOI: 10.13063/2327-9214.1163. View

10.

Cook D, Joyce C, Barnett R, Birgan S, Playford H, Cockings J . Prospective independent validation of APACHE III models in an Australian tertiary adult intensive care unit. Anaesth Intensive Care. 2002; 30(3):308-15. DOI: 10.1177/0310057X0203000307. View

11.

Uchino S, Kellum J, Bellomo R, Doig G, Morimatsu H, Morgera S . Acute renal failure in critically ill patients: a multinational, multicenter study. JAMA. 2005; 294(7):813-8. DOI: 10.1001/jama.294.7.813. View

12.

Amarasingham R, Patzer R, Huesch M, Nguyen N, Xie B . Implementing electronic health care predictive analytics: considerations and challenges. Health Aff (Millwood). 2014; 33(7):1148-54. DOI: 10.1377/hlthaff.2014.0352. View

13.

Steyerberg E, van der Ploeg T, Van Calster B . Risk prediction with machine learning and regression methods. Biom J. 2014; 56(4):601-6. DOI: 10.1002/bimj.201300297. View

14.

Ng S, Sanagou M, Wolfe R, Cochrane A, Smith J, Reid C . Prediction of acute kidney injury within 30 days of cardiac surgery. J Thorac Cardiovasc Surg. 2013; 147(6):1875-83, 1883.e1. DOI: 10.1016/j.jtcvs.2013.06.049. View

15.

Breidthardt T, Christ-Crain M, Stolz D, Bingisser R, Drexler B, Klima T . A combined cardiorenal assessment for the prediction of acute kidney injury in lower respiratory tract infections. Am J Med. 2012; 125(2):168-75. DOI: 10.1016/j.amjmed.2011.07.010. View

16.

Ohno-Machado L, Resnic F, Matheny M . Prognosis in critical care. Annu Rev Biomed Eng. 2006; 8:567-99. DOI: 10.1146/annurev.bioeng.8.061505.095842. View

17.

Parikh R, Kakad M, Bates D . Integrating Predictive Analytics Into High-Value Care: The Dawn of Precision Delivery. JAMA. 2016; 315(7):651-2. DOI: 10.1001/jama.2015.19417. View

18.

Van Hoorde K, Van Huffel S, Timmerman D, Bourne T, Van Calster B . A spline-based tool to assess and visualize the calibration of multiclass risk predictions. J Biomed Inform. 2015; 54:283-93. DOI: 10.1016/j.jbi.2014.12.016. View

19.

Wang Y, Cheng H, Yue T, Chen Y . Derivation and validation of a prediction score for acute kidney injury in patients hospitalized with acute heart failure in a Chinese cohort. Nephrology (Carlton). 2013; 18(7):489-96. DOI: 10.1111/nep.12092. View

20.

Hickey G, Grant S, Murphy G, Bhabra M, Pagano D, McAllister K . Dynamic trends in cardiac surgery: why the logistic EuroSCORE is no longer suitable for contemporary cardiac surgery and implications for future risk models. Eur J Cardiothorac Surg. 2012; 43(6):1146-52. PMC: 3655624. DOI: 10.1093/ejcts/ezs584. View