A Pitfall for Machine Learning Methods Aiming to Predict Across Cell Types

Overview

Journal Genome Biol

Specialties Biology
Genetics

Date 2020 Nov 20

PMID 33213499

Citations 29

Authors

Jacob Schreiber

Ritambhara Singh

Jeffrey Bilmes

William Stafford Noble

Affiliations

Soon will be listed here.

Abstract

Machine learning models that predict genomic activity are most useful when they make accurate predictions across cell types. Here, we show that when the training and test sets contain the same genomic loci, the resulting model may falsely appear to perform well by effectively memorizing the average activity associated with each locus across the training cell types. We demonstrate this phenomenon in the context of predicting gene expression and chromatin domain boundaries, and we suggest methods to diagnose and avoid the pitfall. We anticipate that, as more data becomes available, future projects will increasingly risk suffering from this issue.

Citing Articles

Iterative improvement of deep learning models using synthetic regulatory genomics.

Ribeiro-Dos-Santos A, Maurano M bioRxiv. 2025; .

PMID: 39974895 PMC: 11838587. DOI: 10.1101/2025.02.04.636130.

Loss of MEF2C function by enhancer mutation leads to neuronal mitochondria dysfunction and motor deficits in mice.

Yousefian-Jazi A, Kim S, Chu J, Choi S, Nguyen P, Park U Mol Neurodegener. 2025; 20(1):16.

PMID: 39920775 PMC: 11806887. DOI: 10.1186/s13024-024-00792-y.

Generative modeling for RNA splicing predictions and design.

Wu D, Maus N, Jha A, Yang K, Wales-McGrath B, Jewell S bioRxiv. 2025; .

PMID: 39896553 PMC: 11785043. DOI: 10.1101/2025.01.20.633986.

Best holdout assessment is sufficient for cancer transcriptomic model selection.

Crawford J, Chikina M, Greene C Patterns (N Y). 2025; 5(12):101115.

PMID: 39776849 PMC: 11701843. DOI: 10.1016/j.patter.2024.101115.

Predicting cell type-specific epigenomic profiles accounting for distal genetic effects.

Murphy A, Beardall W, Rei M, Phuycharoen M, Skene N Nat Commun. 2024; 15(1):9951.

PMID: 39550354 PMC: 11569248. DOI: 10.1038/s41467-024-54441-5.

References

Nair S, Kim D, Perricone J, Kundaje A . Integrating regulatory DNA sequence and gene expression to predict genome-wide chromatin accessibility across cellular contexts. Bioinformatics. 2019; 35(14):i108-i116. PMC: 6612838. DOI: 10.1093/bioinformatics/btz352. View

Li Y, Shi W, Wasserman W . Genome-wide prediction of cis-regulatory regions using supervised deep learning methods. BMC Bioinformatics. 2018; 19(1):202. PMC: 5984344. DOI: 10.1186/s12859-018-2187-1. View

Thibodeau A, Uyar A, Khetan S, Stitzel M, Ucar D . A neural network based model effectively predicts enhancers from clinical ATAC-seq samples. Sci Rep. 2018; 8(1):16048. PMC: 6207744. DOI: 10.1038/s41598-018-34420-9. View

Erwin G, Oksenberg N, Truty R, Kostka D, Murphy K, Ahituv N . Integrating diverse datasets improves developmental enhancer prediction. PLoS Comput Biol. 2014; 10(6):e1003677. PMC: 4072507. DOI: 10.1371/journal.pcbi.1003677. View

Singh R, Lanchantin J, Robins G, Qi Y . DeepChrome: deep-learning for predicting gene expression from histone modifications. Bioinformatics. 2016; 32(17):i639-i648. DOI: 10.1093/bioinformatics/btw427. View

Ernst J, Kellis M . Large-scale imputation of epigenomic datasets for systematic annotation of diverse human tissues. Nat Biotechnol. 2015; 33(4):364-76. PMC: 4512306. DOI: 10.1038/nbt.3157. View

. An integrated encyclopedia of DNA elements in the human genome. Nature. 2012; 489(7414):57-74. PMC: 3439153. DOI: 10.1038/nature11247. View

Di Pierro M, Cheng R, Aiden E, Wolynes P, Onuchic J . De novo prediction of human chromosome structures: Epigenetic marking patterns encode genome architecture. Proc Natl Acad Sci U S A. 2017; 114(46):12126-12131. PMC: 5699090. DOI: 10.1073/pnas.1714980114. View

Zhou W, Sherwood B, Ji Z, Xue Y, Du F, Bai J . Genome-wide prediction of DNase I hypersensitivity using gene expression. Nat Commun. 2017; 8(1):1038. PMC: 5715040. DOI: 10.1038/s41467-017-01188-x. View

10.

Ghandi M, Lee D, Mohammad-Noori M, Beer M . Enhanced regulatory sequence prediction using gapped k-mer features. PLoS Comput Biol. 2014; 10(7):e1003711. PMC: 4102394. DOI: 10.1371/journal.pcbi.1003711. View

11.

Durham T, Libbrecht M, Howbert J, Bilmes J, Noble W . PREDICTD PaRallel Epigenomics Data Imputation with Cloud-based Tensor Decomposition. Nat Commun. 2018; 9(1):1402. PMC: 5895786. DOI: 10.1038/s41467-018-03635-9. View

12.

Hoffman M, Buske O, Wang J, Weng Z, Bilmes J, Noble W . Unsupervised pattern discovery in human chromatin structure through genomic segmentation. Nat Methods. 2012; 9(5):473-6. PMC: 3340533. DOI: 10.1038/nmeth.1937. View

13.

Alipanahi B, Delong A, Weirauch M, Frey B . Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning. Nat Biotechnol. 2015; 33(8):831-8. DOI: 10.1038/nbt.3300. View

14.

Kim S, Harwani M, Grama A, Chaterji S . EP-DNN: A Deep Neural Network-Based Global Enhancer Prediction Algorithm. Sci Rep. 2016; 6:38433. PMC: 5144062. DOI: 10.1038/srep38433. View

15.

He Y, Gorkin D, Dickel D, Nery J, Castanon R, Lee A . Improved regulatory element prediction based on tissue-specific local epigenomic signatures. Proc Natl Acad Sci U S A. 2017; 114(9):E1633-E1640. PMC: 5338528. DOI: 10.1073/pnas.1618353114. View

16.

Ernst J, Kellis M . Discovery and characterization of chromatin states for systematic annotation of the human genome. Nat Biotechnol. 2010; 28(8):817-25. PMC: 2919626. DOI: 10.1038/nbt.1662. View

17.

Lu Y, Qu W, Shan G, Zhang C . DELTA: A Distal Enhancer Locating Tool Based on AdaBoost Algorithm and Shape Features of Chromatin Modifications. PLoS One. 2015; 10(6):e0130622. PMC: 4474808. DOI: 10.1371/journal.pone.0130622. View

18.

Fernandez M, Miranda-Saavedra D . Genome-wide enhancer prediction from epigenetic signatures using genetic algorithm-optimized support vector machines. Nucleic Acids Res. 2012; 40(10):e77. PMC: 3378905. DOI: 10.1093/nar/gks149. View

19.

Angermueller C, Lee H, Reik W, Stegle O . DeepCpG: accurate prediction of single-cell DNA methylation states using deep learning. Genome Biol. 2017; 18(1):67. PMC: 5387360. DOI: 10.1186/s13059-017-1189-z. View

20.

Singh R, Lanchantin J, Sekhon A, Qi Y . Attend and Predict: Understanding Gene Regulation by Selective Attention on Chromatin. Adv Neural Inf Process Syst. 2018; 30:6785-6795. PMC: 6105294. View