External Validation of a Prognostic Model for Mortality Among Patients with Non-small-cell Lung Cancer Using the Veterans Precision Oncology Data Commons

Overview

Journal Semin Oncol

Specialty Oncology

Date 2019 Nov 12

PMID 31708233

Citations 2

Authors

David Cheng

Jaime Ramos-Cejudo

David Tuck

Danne Elbers

Mary Brophy

Nhan Do

Nathanael Fillmore

Affiliations

Soon will be listed here.

Abstract

Background: There is wide interest in developing prognostic models in non-small-cell lung cancer (NSCLC) due to the heterogeneity of the disease. Models developed at other healthcare institutions may not be directly applicable for patients treated at the Department of Veterans Affairs (VA). External validation of a candidate prognostic model among VA patients would be crucial before it can be implemented to aid clinical decision-making.

Methods: A prognostic model for mortality developed in the Military Health System (MHS) was applied to data from the VA Precision Oncology Data Repository (VA-PODR), which is available to researchers inside and outside the VA at the Veterans Precision Oncology Data Commons (VPODC). Measures of discrimination and calibration were calculated for the MHS model. The MHS model was also refitted in VA-PODR data using the same risk factors to compare the effect of specific factors and predictive performance when the model is developed using VA data.

Results: Time-dependent AUC of the MHS prognostic model was 0.788, 0.806, 0.780, and 0.779 for predicting survival at 1, 2, 3, and 5 years following diagnosis, respectively. Significant discrepancies were found between predicted and observed rates of survival, particularly for later years. When the model is refit in VA-PODR data, it achieved cross-validated AUCs of 0.739, 0.773, 0.769, and 0.807 at the same time points, and discrepancies between predicted and observed survival were reduced.

Conclusions: Validation of the MHS prognostic model in VA-PODR demonstrates that its discrimination remains strong when applied to VA patients. Nevertheless, further calibration to VA data may be needed to improve its risk estimation performance. This study highlights the utility of VA-PODR and the VPODC as a national resource for developing analytic tools that are well adapted to the Veteran population.

Citing Articles

Building Research Infrastructure to Develop Greater Learning Efficiencies (BRIDGE).

Elbers D, Fillmore N, La J, Tosi H, Ajjarapu S, Dhond R Stud Health Technol Inform. 2024; 310:1131-1135.

PMID: 38269991 PMC: 11166042. DOI: 10.3233/SHTI231141.

The Veterans Affairs Precision Oncology Data Repository, a Clinical, Genomic, and Imaging Research Database.

Elbers D, Fillmore N, Sung F, Ganas S, Prokhorenkov A, Meyer C Patterns (N Y). 2020; 1(6):100083.

PMID: 33205130 PMC: 7660389. DOI: 10.1016/j.patter.2020.100083.

References

Mahar A, Compton C, McShane L, Halabi S, Asamura H, Rami-Porta R . Refining Prognosis in Lung Cancer: A Report on the Quality and Relevance of Clinical Prognostic Tools. J Thorac Oncol. 2015; 10(11):1576-89. PMC: 4636439. DOI: 10.1097/JTO.0000000000000652. View

Dehing-Oberije C, Aerts H, Yu S, De Ruysscher D, Menheere P, Hilvo M . Development and validation of a prognostic model using blood biomarker information for prediction of survival of non-small-cell lung cancer patients treated with combined chemotherapy and radiation or radiotherapy alone (NCT00181519, NCT00573040,.... Int J Radiat Oncol Biol Phys. 2010; 81(2):360-8. DOI: 10.1016/j.ijrobp.2010.06.011. View

Collins G, de Groot J, Dutton S, Omar O, Shanyinde M, Tajar A . External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC Med Res Methodol. 2014; 14:40. PMC: 3999945. DOI: 10.1186/1471-2288-14-40. View

Hoang T, Dahlberg S, Sandler A, Brahmer J, Schiller J, Johnson D . Prognostic models to predict survival in non-small-cell lung cancer patients treated with first-line paclitaxel and carboplatin with or without bevacizumab. J Thorac Oncol. 2012; 7(9):1361-8. PMC: 3433686. DOI: 10.1097/JTO.0b013e318260e106. View

Lin J, Carter C, McGlynn K, Zahm S, Nations J, Anderson W . A Prognostic Model to Predict Mortality among Non-Small-Cell Lung Cancer Patients in the U.S. Military Health System. J Thorac Oncol. 2015; 10(12):1694-702. PMC: 4695230. DOI: 10.1097/JTO.0000000000000691. View

Zullig L, Sims K, McNeil R, Williams C, Jackson G, Provenzale D . Cancer Incidence Among Patients of the U.S. Veterans Affairs Health Care System: 2010 Update. Mil Med. 2017; 182(7):e1883-e1891. PMC: 5650119. DOI: 10.7205/MILMED-D-16-00371. View

Debray T, Koffijberg H, Vergouwe Y, Moons K, Steyerberg E . Aggregating published prediction models with individual participant data: a comparison of different approaches. Stat Med. 2012; 31(23):2697-712. DOI: 10.1002/sim.5412. View

Revesz D, Engelhardt E, Tamminga J, Schramel F, Onwuteaka-Philipsen B, Garde E . Decision support systems for incurable non-small cell lung cancer: a systematic review. BMC Med Inform Decis Mak. 2017; 17(1):144. PMC: 5625762. DOI: 10.1186/s12911-017-0542-1. View

Heagerty P, Lumley T, Pepe M . Time-dependent ROC curves for censored survival data and a diagnostic marker. Biometrics. 2000; 56(2):337-44. DOI: 10.1111/j.0006-341x.2000.00337.x. View

10.

Vergouwe Y, Moons K, Steyerberg E . External validity of risk models: Use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol. 2010; 172(8):971-80. PMC: 2984249. DOI: 10.1093/aje/kwq223. View

11.

Riley R, Ensor J, Snell K, Debray T, Altman D, Moons K . External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges. BMJ. 2016; 353:i3140. PMC: 4916924. DOI: 10.1136/bmj.i3140. View

12.

Medeiros F, Weinreb R, Sample P, Gomi C, Bowd C, Crowston J . Validation of a predictive model to estimate the risk of conversion from ocular hypertension to glaucoma. Arch Ophthalmol. 2005; 123(10):1351-60. DOI: 10.1001/archopht.123.10.1351. View

13.

Park M, Lee J, Hong J, Choi M, Yi J, Lee S . Prognostic model to predict outcomes in nonsmall cell lung cancer patients treated with gefitinib as a salvage treatment. Cancer. 2009; 115(7):1518-30. DOI: 10.1002/cncr.24151. View

14.

Siegel R, Miller K, Jemal A . Cancer statistics, 2018. CA Cancer J Clin. 2018; 68(1):7-30. DOI: 10.3322/caac.21442. View

15.

Fiore L, Ferguson R, Brophy M, Kudesia V, Shannon C, Zimolzak A . Implementation of a Precision Oncology Program as an Exemplar of a Learning Health Care System in the VA. Fed Pract. 2019; 33(Suppl 1):26S-30S. PMC: 6375409. View

16.

Mallett S, Royston P, Waters R, Dutton S, Altman D . Reporting performance of prognostic models in cancer: a review. BMC Med. 2010; 8:21. PMC: 2857810. DOI: 10.1186/1741-7015-8-21. View

17.

Breslow N . Covariance analysis of censored survival data. Biometrics. 1974; 30(1):89-99. View

18.

Flather M, Delahunty N, Collinson J . Generalizing results of randomized trials to clinical practice: reliability and cautions. Clin Trials. 2006; 3(6):508-12. DOI: 10.1177/1740774506073464. View

19.

Yan A, Jong P, Yan R, Tan M, Fitchett D, Chow C . Clinical trial--derived risk model may not generalize to real-world patients with acute coronary syndrome. Am Heart J. 2005; 148(6):1020-7. DOI: 10.1016/j.ahj.2004.02.014. View

20.

Fiore L, Brophy M, Turek S, Kudesia V, Ramnath N, Shannon C . The VA Point-of-Care Precision Oncology Program: Balancing Access with Rapid Learning in Molecular Cancer Medicine. Biomark Cancer. 2016; 8:9-16. PMC: 4772906. DOI: 10.4137/BIC.S37548. View