A Modified Net Reclassification Improvement Statistic

Overview

Journal J Stat Plan Inference

Publisher Elsevier

Date 2023 Apr 10

PMID 37035267

Authors

Glenn Heller

Affiliations

Soon will be listed here.

Abstract

The continuous net reclassification improvement (NRI) statistic is a popular model change measure that was developed to assess the incremental value of new factors in a risk prediction model. Two prominent statistical issues identified in the literature call the utility of this measure into question: (1) it is not a proper scoring function and (2) it has a high false positive rate when testing whether new factors contribute to the risk model. For binary response regression models, these subjects are interrogated and a modification of the continuous NRI, guided by the likelihood-based score residual, is proposed to address these issues. Within a nested model framework, the modified NRI may be viewed as a distance measure between two risk models. An application of the modified NRI is illustrated using prostate cancer data.

References

Pencina M, DAgostino Sr R, DAgostino Jr R, Vasan R . Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med. 2007; 27(2):157-72. DOI: 10.1002/sim.2929. View

Pepe M, Fan J, Feng Z, Gerds T, Hilden J . The Net Reclassification Index (NRI): a Misleading Measure of Prediction Improvement Even with Independent Test Data Sets. Stat Biosci. 2015; 7(2):282-295. PMC: 4615606. DOI: 10.1007/s12561-014-9118-0. View

Cieslikowski W, Antczak A, Nowicki M, Zabel M, Budna-Tukan J . Clinical Relevance of Circulating Tumor Cells in Prostate Cancer Management. Biomedicines. 2021; 9(9). PMC: 8471111. DOI: 10.3390/biomedicines9091179. View

Sayegh N, Swami U, Agarwal N . Recent Advances in the Management of Metastatic Prostate Cancer. JCO Oncol Pract. 2021; 18(1):45-55. DOI: 10.1200/OP.21.00206. View

Kerr K, Wang Z, Janes H, McClelland R, Psaty B, Pepe M . Net reclassification indices for evaluating risk prediction instruments: a critical review. Epidemiology. 2013; 25(1):114-21. PMC: 3918180. DOI: 10.1097/EDE.0000000000000018. View

Gafita A, Calais J, Grogan T, Hadaschik B, Wang H, Weber M . Nomograms to predict outcomes after Lu-PSMA therapy in men with metastatic castration-resistant prostate cancer: an international, multicentre, retrospective study. Lancet Oncol. 2021; 22(8):1115-1125. DOI: 10.1016/S1470-2045(21)00274-6. View

Pepe M, Kerr K, Longton G, Wang Z . Testing for improvement in prediction model performance. Stat Med. 2013; 32(9):1467-82. PMC: 3625503. DOI: 10.1002/sim.5727. View

Pencina M, DAgostino Sr R, Steyerberg E . Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Stat Med. 2011; 30(1):11-21. PMC: 3341973. DOI: 10.1002/sim.4085. View

Pepe M, Janes H, Li C . Net risk reclassification p values: valid or misleading?. J Natl Cancer Inst. 2014; 106(4):dju041. PMC: 3982889. DOI: 10.1093/jnci/dju041. View

10.

Saad F, Fizazi K, Jinga V, Efstathiou E, Fong P, Hart L . Orteronel plus prednisone in patients with chemotherapy-naive metastatic castration-resistant prostate cancer (ELM-PC 4): a double-blind, multicentre, phase 3, randomised, placebo-controlled trial. Lancet Oncol. 2015; 16(3):338-48. DOI: 10.1016/S1470-2045(15)70027-6. View

11.

Demler O, Pencina M, Cook N, DAgostino Sr R . Asymptotic distribution of ∆AUC, NRIs, and IDI based on theory of U-statistics. Stat Med. 2017; 36(21):3334-3360. PMC: 5931715. DOI: 10.1002/sim.7333. View

12.

Pencina M, DAgostino Sr R, Demler O . Novel metrics for evaluating improvement in discrimination: net reclassification and integrated discrimination improvement for normal variables and nested models. Stat Med. 2011; 31(2):101-13. PMC: 3341978. DOI: 10.1002/sim.4348. View

13.

Kerr K, McClelland R, Brown E, Lumley T . Evaluating the incremental value of new biomarkers with integrated discrimination improvement. Am J Epidemiol. 2011; 174(3):364-74. PMC: 3202159. DOI: 10.1093/aje/kwr086. View

14.

Ryan C, Dutta S, Kelly W, Russell C, Small E, Morris M . Androgen decline and survival during docetaxel therapy in metastatic castration resistant prostate cancer (mCRPC). Prostate Cancer Prostatic Dis. 2019; 23(1):66-73. PMC: 6825875. DOI: 10.1038/s41391-019-0152-3. View