Preventing Dataset Shift from Breaking Machine-learning Biomarkers

Overview

Journal Gigascience

Publisher Oxford University Press

Specialties Biology
Genetics

Date 2021 Sep 29

PMID 34585237

Citations 23

Authors

Jerome Dockes

Gael Varoquaux

Jean-Baptiste Poline

Affiliations

Soon will be listed here.

Abstract

Machine learning brings the hope of finding new biomarkers extracted from cohorts with rich biomedical measurements. A good biomarker is one that gives reliable detection of the corresponding condition. However, biomarkers are often extracted from a cohort that differs from the target population. Such a mismatch, known as a dataset shift, can undermine the application of the biomarker to new individuals. Dataset shifts are frequent in biomedical research, e.g., because of recruitment biases. When a dataset shift occurs, standard machine-learning techniques do not suffice to extract and validate biomarkers. This article provides an overview of when and how dataset shifts break machine-learning-extracted biomarkers, as well as detection and correction strategies.

Citing Articles

Transferability of Single- and Cross-Tissue Transcriptome Imputation Models Across Ancestry Groups.

Pagnuco I, Eyre S, Rattray M, Morris A Genet Epidemiol. 2025; 49(1):e22611.

PMID: 39812501 PMC: 11734644. DOI: 10.1002/gepi.22611.

Magnetic Resonance Imaging Liver Segmentation Protocol Enables More Consistent and Robust Annotations, Paving the Way for Advanced Computer-Assisted Analysis.

Jeltsch P, Monnin K, Jreige M, Fernandes-Mendes L, Girardet R, Dromain C Diagnostics (Basel). 2025; 14(24.

PMID: 39767146 PMC: 11726866. DOI: 10.3390/diagnostics14242785.

Power and reproducibility in the external validation of brain-phenotype predictions.

Rosenblatt M, Tejavibulya L, Sun H, Camp C, Khaitova M, Adkinson B Nat Hum Behav. 2024; 8(10):2018-2033.

PMID: 39085406 DOI: 10.1038/s41562-024-01931-7.

Considerations for Quality Control Monitoring of Machine Learning Models in Clinical Practice.

Faust L, Wilson P, Asai S, Fu S, Liu H, Ruan X JMIR Med Inform. 2024; 12:e50437.

PMID: 38941140 PMC: 11245651. DOI: 10.2196/50437.

Pathophysiological Features in Electronic Medical Records Sustain Model Performance under Temporal Dataset Shift.

Brosula R, Corbin C, Chen J AMIA Jt Summits Transl Sci Proc. 2024; 2024:95-104.

PMID: 38827052 PMC: 11141811.

References

Dockes J, Varoquaux G, Poline J . Preventing dataset shift from breaking machine-learning biomarkers. Gigascience. 2021; 10(9). PMC: 8478611. DOI: 10.1093/gigascience/giab055. View

Henrich J, Heine S, Norenzayan A . Most people are not WEIRD. Nature. 2010; 466(7302):29. DOI: 10.1038/466029a. View

Faust O, Hagiwara Y, Hong T, Lih O, Rajendra Acharya U . Deep learning for healthcare applications based on physiological signals: A review. Comput Methods Programs Biomed. 2018; 161:1-13. DOI: 10.1016/j.cmpb.2018.04.005. View

Hernan M, Hernandez-Diaz S, Robins J . A structural approach to selection bias. Epidemiology. 2004; 15(5):615-25. DOI: 10.1097/01.ede.0000135174.63482.43. View

Adamson A, Smith A . Machine Learning and Health Care Disparities in Dermatology. JAMA Dermatol. 2018; 154(11):1247-1248. DOI: 10.1001/jamadermatol.2018.2348. View

Little M, Varoquaux G, Saeb S, Lonini L, Jayaraman A, Mohr D . Using and understanding cross-validation strategies. Perspectives on Saeb et al. Gigascience. 2017; 6(5):1-6. PMC: 5441396. DOI: 10.1093/gigascience/gix020. View

Kasahara R, Kino S, Soyama S, Matsuura Y . Noninvasive glucose monitoring using mid-infrared absorption spectroscopy based on a few wavenumbers. Biomed Opt Express. 2018; 9(1):289-302. PMC: 5772583. DOI: 10.1364/BOE.9.000289. View

Larrazabal A, Nieto N, Peterson V, Milone D, Ferrante E . Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis. Proc Natl Acad Sci U S A. 2020; 117(23):12592-12594. PMC: 7293650. DOI: 10.1073/pnas.1919012117. View

Gianfrancesco M, Tamang S, Yazdany J, Schmajuk G . Potential Biases in Machine Learning Algorithms Using Electronic Health Record Data. JAMA Intern Med. 2018; 178(11):1544-1547. PMC: 6347576. DOI: 10.1001/jamainternmed.2018.3763. View

10.

Bleeker S, Moll H, Steyerberg E, Donders A, Derksen-Lubsen G, Grobbee D . External validation is necessary in prediction research: a clinical example. J Clin Epidemiol. 2003; 56(9):826-32. DOI: 10.1016/s0895-4356(03)00207-5. View

11.

Woo C, Chang L, Lindquist M, Wager T . Building better biomarkers: brain models in translational neuroimaging. Nat Neurosci. 2017; 20(3):365-377. PMC: 5988350. DOI: 10.1038/nn.4478. View

12.

Saez C, Gutierrez-Sacristan A, Kohane I, Garcia-Gomez J, Avillach P . EHRtemporalVariability: delineating temporal data-set shifts in electronic health records. Gigascience. 2020; 9(8). PMC: 7391413. DOI: 10.1093/gigascience/giaa079. View

13.

Mulherin S, Miller W . Spectrum bias or spectrum effect? Subgroup variation in diagnostic test evaluation. Ann Intern Med. 2002; 137(7):598-602. DOI: 10.7326/0003-4819-137-7-200210010-00011. View

14.

Heiat A, Gross C, Krumholz H . Representation of the elderly, women, and minorities in heart failure clinical trials. Arch Intern Med. 2002; 162(15):1682-8. DOI: 10.1001/archinte.162.15.1682. View

15.

Ransohoff D, Feinstein A . Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. N Engl J Med. 1978; 299(17):926-30. DOI: 10.1056/NEJM197810262991705. View

16.

Deo R . Machine Learning in Medicine. Circulation. 2015; 132(20):1920-30. PMC: 5831252. DOI: 10.1161/CIRCULATIONAHA.115.001593. View

17.

Oakden-Rayner L, Dunnmon J, Carneiro G, Re C . Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imaging. Proc ACM Conf Health Inference Learn (2020). 2020; 2020:151-159. PMC: 7665161. DOI: 10.1145/3368555.3384468. View

18.

Kouw W, Loog M . A Review of Domain Adaptation without Target Labels. IEEE Trans Pattern Anal Mach Intell. 2019; 43(3):766-785. DOI: 10.1109/TPAMI.2019.2945942. View

19.

Chastain D, Osae S, Henao-Martinez A, Franco-Paredes C, Chastain J, Young H . Racial Disproportionality in Covid Clinical Trials. N Engl J Med. 2020; 383(9):e59. DOI: 10.1056/NEJMp2021971. View

20.

Wynants L, Van Calster B, Collins G, Riley R, Heinze G, Schuit E . Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ. 2020; 369:m1328. PMC: 7222643. DOI: 10.1136/bmj.m1328. View