Empirical Evaluation of Sub-cohort Sampling Designs for Risk Prediction Modeling

Sub-cohort sampling designs, such as nested case-control (NCC) and case-cohort (CC) studies, have been widely used to estimate biomarker-disease associations because of their cost effectiveness. These designs have been well studied and shown to maintain relatively high efficiency compared to full-cohort designs, but their performance of building risk prediction models has been less studied. Moreover, sub-cohort sampling designs often use matching (or stratifying) to further control for confounders or to reduce measurement error. Their predictive performance depends on both the design and matching procedures. Based on a dataset from the NYU Women's Health Study (NYUWHS), we performed Monte Carlo simulations to systematically evaluate risk prediction performance under NCC, CC, and full-cohort studies. Our simulations demonstrate that sub-cohort sampling designs can have predictive accuracy (i.e. discrimination and calibration) similar to that of the full-cohort design, but could be sensitive to the matching procedure used. Our results suggest that researchers can have the option of performing NCC and CC studies with huge potential benefits in cost and resources, but need to pay particular attention to the matching procedure when developing a risk prediction model in biomarker studies.