» Articles » PMID: 35741566

Using Background Knowledge from Preceding Studies for Building a Random Forest Prediction Model: A Plasmode Simulation Study

Overview
Journal Entropy (Basel)
Publisher MDPI
Date 2022 Jun 24
PMID 35741566
Authors
Affiliations
Soon will be listed here.
Abstract

There is an increasing interest in machine learning (ML) algorithms for predicting patient outcomes, as these methods are designed to automatically discover complex data patterns. For example, the random forest (RF) algorithm is designed to identify relevant predictor variables out of a large set of candidates. In addition, researchers may also use external information for variable selection to improve model interpretability and variable selection accuracy, thereby prediction quality. However, it is unclear to which extent, if at all, RF and ML methods may benefit from external information. In this paper, we examine the usefulness of external information from prior variable selection studies that used traditional statistical modeling approaches such as the Lasso, or suboptimal methods such as univariate selection. We conducted a plasmode simulation study based on subsampling a data set from a pharmacoepidemiologic study with nearly 200,000 individuals, two binary outcomes and 1152 candidate predictor (mainly sparse binary) variables. When the scope of candidate predictors was reduced based on external knowledge RF models achieved better calibration, that is, better agreement of predictions and observed outcome rates. However, prediction quality measured by cross-entropy, AUROC or the Brier score did not improve. We recommend appraising the methodological quality of studies that serve as an external information source for future prediction model development.

References
1.
Sauerbrei W, Perperoglou A, Schmid M, Abrahamowicz M, Becher H, Binder H . State of the art in selection of variables and functional forms in multivariable analysis-outstanding issues. Diagn Progn Res. 2020; 4:3. PMC: 7114804. DOI: 10.1186/s41512-020-00074-3. View

2.
Friedman J, Hastie T, Tibshirani R . Regularization Paths for Generalized Linear Models via Coordinate Descent. J Stat Softw. 2010; 33(1):1-22. PMC: 2929880. View

3.
Malley J, Kruppa J, Dasgupta A, Malley K, Ziegler A . Probability machines: consistent probability estimation using nonparametric learning machines. Methods Inf Med. 2011; 51(1):74-81. PMC: 3250568. DOI: 10.3414/ME00-01-0052. View

4.
Sun G, Shook T, Kay G . Inappropriate use of bivariable analysis to screen risk factors for use in multivariable analysis. J Clin Epidemiol. 1996; 49(8):907-16. DOI: 10.1016/0895-4356(96)00025-x. View

5.
Moons K, Wolff R, Riley R, Whiting P, Westwood M, Collins G . PROBAST: A Tool to Assess Risk of Bias and Applicability of Prediction Model Studies: Explanation and Elaboration. Ann Intern Med. 2019; 170(1):W1-W33. DOI: 10.7326/M18-1377. View