Development and Validation of Self-monitoring Auto-updating Prognostic Models of Survival for Hospitalized COVID-19 Patients

Overview

Journal Nat Commun

Specialty Biology

Date 2022 Nov 10

PMID 36357420

Authors

Todd J Levy

Kevin Coppa

Jinxuan Cang

Douglas P Barnaby

Marc D Paradis

Stuart L Cohen

Alex Makhnevich

David van Klaveren

David M Kent

Karina W Davidson

Jamie S Hirsch

Theodoros P Zanos

Affiliations

Soon will be listed here.

Abstract

Clinical prognostic models can assist patient care decisions. However, their performance can drift over time and location, necessitating model monitoring and updating. Despite rapid and significant changes during the pandemic, prognostic models for COVID-19 patients do not currently account for these drifts. We develop a framework for continuously monitoring and updating prognostic models and apply it to predict 28-day survival in COVID-19 patients. We use demographic, laboratory, and clinical data from electronic health records of 34912 hospitalized COVID-19 patients from March 2020 until May 2022 and compare three modeling methods. Model calibration performance drift is immediately detected with minor fluctuations in discrimination. The overall calibration on the prospective validation cohort is significantly improved when comparing the dynamically updated models against their static counterparts. Our findings suggest that, using this framework, models remain accurate and well-calibrated across various waves, variants, race and sex and yield positive net-benefits.

Citing Articles

Monitoring performance of clinical artificial intelligence in health care: a scoping review.

Andersen E, Birk-Korch J, Hansen R, Fly L, Rottger R, Arcani D JBI Evid Synth. 2024; 22(12):2423-2446.

PMID: 39658865 PMC: 11630661. DOI: 10.11124/JBIES-24-00042.

European and US Guideline-Based Statin Eligibility, Genetically Predicted Coronary Artery Disease, and the Risk of Major Coronary Events.

Park H, Kim D, Chan You S, Jang E, Yu H, Kim T J Am Heart Assoc. 2024; 13(9):e032831.

PMID: 38639378 PMC: 11179899. DOI: 10.1161/JAHA.123.032831.

Empirical data drift detection experiments on real-world medical imaging data.

Kore A, Abbasi Bavil E, Subasri V, Abdalla M, Fine B, Dolatabadi E Nat Commun. 2024; 15(1):1887.

PMID: 38424096 PMC: 10904813. DOI: 10.1038/s41467-024-46142-w.

Seeing the random forest through the decision trees. Supporting learning health systems from histopathology with machine learning models: Challenges and opportunities.

Gonzalez R, Saha A, Campbell C, Nejat P, Lokker C, Norgan A J Pathol Inform. 2024; 15:100347.

PMID: 38162950 PMC: 10755052. DOI: 10.1016/j.jpi.2023.100347.

Longitudinal dynamic clinical phenotypes of in-hospital COVID-19 patients across three dominant virus variants in New York.

Ho M, Levy T, Koulas I, Founta K, Coppa K, Hirsch J Int J Med Inform. 2023; 181:105286.

PMID: 37956643 PMC: 10843635. DOI: 10.1016/j.ijmedinf.2023.105286.

References

Nashef S, Roques F, Sharples L, Nilsson J, Smith C, Goldstone A . EuroSCORE II. Eur J Cardiothorac Surg. 2012; 41(4):734-44. DOI: 10.1093/ejcts/ezs043. View

Gulati G, Upshaw J, Wessler B, Brazil R, Nelson J, van Klaveren D . Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models. Circ Cardiovasc Qual Outcomes. 2022; 15(4):e008487. PMC: 9015037. DOI: 10.1161/CIRCOUTCOMES.121.008487. View

Vickers A, Cronin A, Elkin E, Gonen M . Extensions to decision curve analysis, a novel method for evaluating diagnostic tests, prediction models and molecular markers. BMC Med Inform Decis Mak. 2008; 8:53. PMC: 2611975. DOI: 10.1186/1472-6947-8-53. View

Horwitz L, Jones S, Cerfolio R, Francois F, Greco J, Rudy B . Trends in COVID-19 Risk-Adjusted Mortality Rates. J Hosp Med. 2020; 16(2):90-92. DOI: 10.12788/jhm.3552. View

Hippisley-Cox J, Coupland C, Brindle P . Development and validation of QRISK3 risk prediction algorithms to estimate future risk of cardiovascular disease: prospective cohort study. BMJ. 2017; 357:j2099. PMC: 5441081. DOI: 10.1136/bmj.j2099. View

Davis S, Greevy Jr R, Lasko T, Walsh C, Matheny M . Detection of calibration drift in clinical prediction models to inform model updating. J Biomed Inform. 2020; 112:103611. PMC: 8627243. DOI: 10.1016/j.jbi.2020.103611. View

Haybittle J, Blamey R, Elston C, Johnson J, DOYLE P, Campbell F . A prognostic index in primary breast cancer. Br J Cancer. 1982; 45(3):361-6. PMC: 2010939. DOI: 10.1038/bjc.1982.62. View

Lentner J, Adams T, Knutson V, Zeien S, Abbas H, Moosavi R . C-reactive protein levels associated with COVID-19 outcomes in the United States. J Osteopath Med. 2021; 121(12):869-873. DOI: 10.1515/jom-2021-0103. View

Nashef S, Roques F, Michel P, Gauducheau E, Lemeshow S, Salamon R . European system for cardiac operative risk evaluation (EuroSCORE). Eur J Cardiothorac Surg. 1999; 16(1):9-13. DOI: 10.1016/s1010-7940(99)00134-7. View

10.

Jarman B, Pieter D, van der Veen A, Kool R, Aylin P, Bottle A . The hospital standardised mortality ratio: a powerful tool for Dutch hospitals to assess their quality of care?. Qual Saf Health Care. 2010; 19(1):9-13. PMC: 2921266. DOI: 10.1136/qshc.2009.032953. View

11.

Dennis J, McGovern A, Vollmer S, Mateen B . Improving Survival of Critical Care Patients With Coronavirus Disease 2019 in England: A National Cohort Study, March to June 2020. Crit Care Med. 2020; 49(2):209-214. PMC: 7803441. DOI: 10.1097/CCM.0000000000004747. View

12.

Wessler B, Nelson J, Park J, McGinnes H, Gulati G, Brazil R . External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature. Circ Cardiovasc Qual Outcomes. 2021; 14(8):e007858. PMC: 8366535. DOI: 10.1161/CIRCOUTCOMES.121.007858. View

13.

Freeman A, Watson A, ORegan P, Wysocki O, Burke H, Freitas A . Wave comparisons of clinical characteristics and outcomes of COVID-19 admissions - Exploring the impact of treatment and strain dynamics. J Clin Virol. 2021; 146:105031. PMC: 8608665. DOI: 10.1016/j.jcv.2021.105031. View

14.

Hippisley-Cox J, Coupland C, Vinogradova Y, Robson J, May M, Brindle P . Derivation and validation of QRISK, a new cardiovascular disease risk score for the United Kingdom: prospective open cohort study. BMJ. 2007; 335(7611):136. PMC: 1925200. DOI: 10.1136/bmj.39261.471806.55. View

15.

Lim W, van der Eerden M, Laing R, Boersma W, Karalus N, Town G . Defining community acquired pneumonia severity on presentation to hospital: an international derivation and validation study. Thorax. 2003; 58(5):377-82. PMC: 1746657. DOI: 10.1136/thorax.58.5.377. View

16.

Jenkins D, Sperrin M, Martin G, Peek N . Dynamic models to predict health outcomes: current status and methodological challenges. Diagn Progn Res. 2019; 2:23. PMC: 6460710. DOI: 10.1186/s41512-018-0045-2. View

17.

Jones A, Trzeciak S, Kline J . The Sequential Organ Failure Assessment score for predicting outcome in patients with severe sepsis and evidence of hypoperfusion at the time of emergency department presentation. Crit Care Med. 2009; 37(5):1649-54. PMC: 2703722. DOI: 10.1097/CCM.0b013e31819def97. View

18.

Hoogland J, van Barreveld M, Debray T, Reitsma J, Verstraelen T, Dijkgraaf M . Handling missing predictor values when validating and applying a prediction model to new patients. Stat Med. 2020; 39(25):3591-3607. PMC: 7586995. DOI: 10.1002/sim.8682. View

19.

Schnellinger E, Yang W, Kimmel S . Comparison of dynamic updating strategies for clinical prediction models. Diagn Progn Res. 2021; 5(1):20. PMC: 8647501. DOI: 10.1186/s41512-021-00110-w. View

20.

Steyerberg E, Moons K, van der Windt D, Hayden J, Perel P, Schroter S . Prognosis Research Strategy (PROGRESS) 3: prognostic model research. PLoS Med. 2013; 10(2):e1001381. PMC: 3564751. DOI: 10.1371/journal.pmed.1001381. View