Transformer Architecture and Attention Mechanisms in Genome Data Analysis: A Comprehensive Review

Overview

Journal Biology (Basel)

Publisher MDPI

Specialty Biology

Date 2023 Jul 29

PMID 37508462

Authors

Sanghyuk Roy Choi

Minhyeok Lee

Affiliations

Soon will be listed here.

Abstract

The emergence and rapid development of deep learning, specifically transformer-based architectures and attention mechanisms, have had transformative implications across several domains, including bioinformatics and genome data analysis. The analogous nature of genome sequences to language texts has enabled the application of techniques that have exhibited success in fields ranging from natural language processing to genomic data. This review provides a comprehensive analysis of the most recent advancements in the application of transformer architectures and attention mechanisms to genome and transcriptome data. The focus of this review is on the critical evaluation of these techniques, discussing their advantages and limitations in the context of genome data analysis. With the swift pace of development in deep learning methodologies, it becomes vital to continually assess and reflect on the current standing and future direction of the research. Therefore, this review aims to serve as a timely resource for both seasoned researchers and newcomers, offering a panoramic view of the recent advancements and elucidating the state-of-the-art applications in the field. Furthermore, this review paper serves to highlight potential areas of future investigation by critically evaluating studies from 2019 to 2023, thereby acting as a stepping-stone for further research endeavors.

Citing Articles

Applications of Artificial Intelligence, Deep Learning, and Machine Learning to Support the Analysis of Microscopic Images of Cells and Tissues.

Ali M, Benfante V, Basirinia G, Alongi P, Sperandeo A, Quattrocchi A J Imaging. 2025; 11(2).

PMID: 39997561 PMC: 11856378. DOI: 10.3390/jimaging11020059.

Sudden Fall Detection of Human Body Using Transformer Model.

Kibet D, So M, Kang H, Han Y, Shin J Sensors (Basel). 2025; 24(24.

PMID: 39771788 PMC: 11679820. DOI: 10.3390/s24248051.

Deep learning-based metabolomics data study of prostate cancer.

Sun L, Fan X, Zhao Y, Zhang Q, Jiang M BMC Bioinformatics. 2024; 25(1):391.

PMID: 39725937 PMC: 11674358. DOI: 10.1186/s12859-024-06016-w.

Opportunities, challenges and future perspectives of using bioinformatics and artificial intelligence techniques on tropical disease identification using omics data.

Vidanagamachchi S, Waidyarathna K Front Digit Health. 2024; 6:1471200.

PMID: 39654982 PMC: 11625773. DOI: 10.3389/fdgth.2024.1471200.

CalTrig: A GUI-based Machine Learning Approach for Decoding Neuronal Calcium Transients in Freely Moving Rodents.

Lange M, Chen Y, Fu H, Korada A, Guo C, Ma Y bioRxiv. 2024; .

PMID: 39372793 PMC: 11451592. DOI: 10.1101/2024.09.30.615860.

References

Raad J, Bugnon L, Milone D, Stegmayer G . miRe2e: a full end-to-end deep model based on transformers for prediction of pre-miRNAs. Bioinformatics. 2021; 38(5):1191-1197. DOI: 10.1093/bioinformatics/btab823. View

Manica M, Oskooei A, Born J, Subramanian V, Saez-Rodriguez J, Rodriguez Martinez M . Toward Explainable Anticancer Compound Sensitivity Prediction via Multimodal Attention-Based Convolutional Encoders. Mol Pharm. 2019; 16(12):4797-4806. DOI: 10.1021/acs.molpharmaceut.9b00520. View

Xiao L, Wan Y, Jiang Z . AttCRISPR: a spacetime interpretable model for prediction of sgRNA on-target activity. BMC Bioinformatics. 2021; 22(1):589. PMC: 8667445. DOI: 10.1186/s12859-021-04509-6. View

Du Z, Xiao X, Uversky V . DeepA-RBPBS: A hybrid convolution and recurrent neural network combined with attention mechanism for predicting RBP binding site. J Biomol Struct Dyn. 2020; 40(9):4250-4258. DOI: 10.1080/07391102.2020.1854861. View

Dominic N, Cenggoro T, Budiarto A, Pardamean B . Deep polygenic neural network for predicting and identifying yield-associated genes in Indonesian rice accessions. Sci Rep. 2022; 12(1):13823. PMC: 9378700. DOI: 10.1038/s41598-022-16075-9. View

Sheng N, Cui H, Zhang T, Xuan P . Attentional multi-level representation encoding based on convolutional and variance autoencoders for lncRNA-disease association prediction. Brief Bioinform. 2020; 22(3). DOI: 10.1093/bib/bbaa067. View

Cao L, Zhang Q, Song H, Lin K, Pang E . DeepASmRNA: Reference-free prediction of alternative splicing events with a scalable and interpretable deep learning model. iScience. 2022; 25(11):105345. PMC: 9619290. DOI: 10.1016/j.isci.2022.105345. View

Kim Y, Lee M . Deep Learning Approaches for lncRNA-Mediated Mechanisms: A Comprehensive Review of Recent Developments. Int J Mol Sci. 2023; 24(12). PMC: 10299203. DOI: 10.3390/ijms241210299. View

Lee M . Recent Advances in Deep Learning for Protein-Protein Interaction Analysis: A Comprehensive Review. Molecules. 2023; 28(13). PMC: 10343845. DOI: 10.3390/molecules28135169. View

10.

Gao Y, Chen Y, Feng H, Zhang Y, Yue Z . RicENN: Prediction of Rice Enhancers with Neural Network Based on DNA Sequences. Interdiscip Sci. 2022; 14(2):555-565. DOI: 10.1007/s12539-022-00503-5. View

11.

Kim E, Nam H . DeSIDE-DDI: interpretable prediction of drug-drug interactions using drug-induced gene expressions. J Cheminform. 2022; 14(1):9. PMC: 8895921. DOI: 10.1186/s13321-022-00589-5. View

12.

Tang X, Zheng P, Li X, Wu H, Wei D, Liu Y . Deep6mAPred: A CNN and Bi-LSTM-based deep learning method for predicting DNA N6-methyladenosine sites across plant species. Methods. 2022; 204:142-150. DOI: 10.1016/j.ymeth.2022.04.011. View

13.

Wekesa J, Meng J, Luan Y . A deep learning model for plant lncRNA-protein interaction prediction with graph attention. Mol Genet Genomics. 2020; 295(5):1091-1102. DOI: 10.1007/s00438-020-01682-w. View

14.

Gomes R, Paul N, He N, Huber A, Jansen R . Application of Feature Selection and Deep Learning for Cancer Prediction Using DNA Methylation Markers. Genes (Basel). 2022; 13(9). PMC: 9498757. DOI: 10.3390/genes13091557. View

15.

Clauwaert J, Menschaert G, Waegeman W . Explainability in transformer models for functional genomics. Brief Bioinform. 2021; 22(5). PMC: 8425421. DOI: 10.1093/bib/bbab060. View

16.

Lee M . Deep Learning Techniques with Genomic Data in Cancer Prognosis: A Comprehensive Review of the 2021-2023 Literature. Biology (Basel). 2023; 12(7). PMC: 10376033. DOI: 10.3390/biology12070893. View

17.

Kasgari A, Safavi S, Nouri M, Hou J, Sarshar N, Ranjbarzadeh R . Point-of-Interest Preference Model Using an Attention Mechanism in a Convolutional Neural Network. Bioengineering (Basel). 2023; 10(4). PMC: 10135568. DOI: 10.3390/bioengineering10040495. View

18.

Yin C, Chen Z . Developing Sustainable Classification of Diseases via Deep Learning and Semi-Supervised Learning. Healthcare (Basel). 2020; 8(3). PMC: 7551840. DOI: 10.3390/healthcare8030291. View

19.

Park S, Koh Y, Jeon H, Kim H, Yeo Y, Kang J . Enhancing the interpretability of transcription factor binding site prediction using attention mechanism. Sci Rep. 2020; 10(1):13413. PMC: 7414127. DOI: 10.1038/s41598-020-70218-4. View

20.

Liu T, Zou B, He M, Hu Y, Dou Y, Cui T . LncReader: identification of dual functional long noncoding RNAs using a multi-head self-attention mechanism. Brief Bioinform. 2022; 24(1). DOI: 10.1093/bib/bbac579. View