Multimodality Fusion Aspects of Medical Diagnosis: A Comprehensive Review

Overview

Journal Bioengineering (Basel)

Date 2025 Jan 8

PMID 39768051

Authors

Sachin Kumar

Sita Rani

Shivani Sharma

Hong Min

Affiliations

Soon will be listed here.

Abstract

Utilizing information from multiple sources is a preferred and more precise method for medical experts to confirm a diagnosis. Each source provides critical information about the disease that might otherwise be absent in other modalities. Combining information from various medical sources boosts confidence in the diagnosis process, enabling the creation of an effective treatment plan for the patient. The scarcity of medical experts to diagnose diseases motivates the development of automatic diagnoses relying on multimodal data. With the progress in artificial intelligence technology, automated diagnosis using multimodal fusion techniques is now possible. Nevertheless, the concept of multimodal medical diagnosis is still new and requires an understanding of the diverse aspects of multimodal data and its related challenges. This review article examines the various aspects of multimodal medical diagnosis to equip readers, academicians, and researchers with necessary knowledge to advance multimodal medical research. The chosen articles in the study underwent thorough screening from reputable journals and publishers to offer high-quality content to readers, who can then apply the knowledge to produce quality research. Besides, the need for multimodal information and the associated challenges are discussed with solutions. Additionally, ethical issues of using artificial intelligence in medical diagnosis is also discussed.

References

Suk H, Lee S, Shen D . Hierarchical feature representation and multimodal fusion with deep learning for AD/MCI diagnosis. Neuroimage. 2014; 101:569-82. PMC: 4165842. DOI: 10.1016/j.neuroimage.2014.06.077. View

Ou C, Zhou S, Yang R, Jiang W, He H, Gan W . A deep learning based multimodal fusion model for skin lesion diagnosis using smartphone collected clinical images and metadata. Front Surg. 2022; 9:1029991. PMC: 9577400. DOI: 10.3389/fsurg.2022.1029991. View

Gao J, Li P, Chen Z, Zhang J . A Survey on Deep Learning for Multimodal Data Fusion. Neural Comput. 2020; 32(5):829-864. DOI: 10.1162/neco_a_01273. View

Amal S, Safarnejad L, Omiye J, Ghanzouri I, Cabot J, Ross E . Use of Multi-Modal Data and Machine Learning to Improve Cardiovascular Disease Care. Front Cardiovasc Med. 2022; 9:840262. PMC: 9091962. DOI: 10.3389/fcvm.2022.840262. View

Mukerji S, Petersen K, Pohl K, Dastgheyb R, Fox H, Bilder R . Machine Learning Approaches to Understand Cognitive Phenotypes in People With HIV. J Infect Dis. 2023; 227(Suppl 1):S48-S57. PMC: 10022709. DOI: 10.1093/infdis/jiac293. View

Steyaert S, Pizurica M, Nagaraj D, Khandelwal P, Hernandez-Boussard T, Gentles A . Multimodal data fusion for cancer biomarker discovery with deep learning. Nat Mach Intell. 2023; 5(4):351-362. PMC: 10484010. DOI: 10.1038/s42256-023-00633-5. View

Zhou H, Yu Y, Wang C, Zhang S, Gao Y, Pan J . A transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics. Nat Biomed Eng. 2023; 7(6):743-755. DOI: 10.1038/s41551-023-01045-x. View

Zhang T, Shi M . Multi-modal neuroimaging feature fusion for diagnosis of Alzheimer's disease. J Neurosci Methods. 2020; 341:108795. DOI: 10.1016/j.jneumeth.2020.108795. View

Hao Y, Jing X, Sun Q . Cancer survival prediction by learning comprehensive deep feature representation for multiple types of genetic data. BMC Bioinformatics. 2023; 24(1):267. PMC: 10308712. DOI: 10.1186/s12859-023-05392-z. View

10.

Sertkaya A, Wong H, Jessup A, Beleche T . Key cost drivers of pharmaceutical clinical trials in the United States. Clin Trials. 2016; 13(2):117-26. DOI: 10.1177/1740774515625964. View

11.

Xu Y, Lin Y, Bell R, Towe S, Pearson J, Nadeem T . Machine learning prediction of neurocognitive impairment among people with HIV using clinical and multimodal magnetic resonance imaging data. J Neurovirol. 2021; 27(1):1-11. PMC: 8001877. DOI: 10.1007/s13365-020-00930-4. View

12.

Carrillo-Perez F, Morales J, Castillo-Secilla D, Gevaert O, Rojas I, Herrera L . Machine-Learning-Based Late Fusion on Multi-Omics and Multi-Scale Data for Non-Small-Cell Lung Cancer Diagnosis. J Pers Med. 2022; 12(4). PMC: 9025878. DOI: 10.3390/jpm12040601. View

13.

Steyaert S, Qiu Y, Zheng Y, Mukherjee P, Vogel H, Gevaert O . Multimodal deep learning to predict prognosis in adult and pediatric brain tumors. Commun Med (Lond). 2023; 3(1):44. PMC: 10060397. DOI: 10.1038/s43856-023-00276-y. View

14.

Xie W, Wang C, Lin Z, Luo X, Chen W, Xu M . Multimodal fusion diagnosis of depression and anxiety based on CNN-LSTM model. Comput Med Imaging Graph. 2022; 102:102128. DOI: 10.1016/j.compmedimag.2022.102128. View

15.

Boehm K, Aherne E, Ellenson L, Nikolovski I, Alghamdi M, Vazquez-Garcia I . Multimodal data integration using machine learning improves risk stratification of high-grade serous ovarian cancer. Nat Cancer. 2022; 3(6):723-733. PMC: 9239907. DOI: 10.1038/s43018-022-00388-9. View

16.

Huang S, Pareek A, Seyyedi S, Banerjee I, Lungren M . Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. NPJ Digit Med. 2020; 3:136. PMC: 7567861. DOI: 10.1038/s41746-020-00341-z. View

17.

Vale-Silva L, Rohr K . Long-term cancer survival prediction using multimodal deep learning. Sci Rep. 2021; 11(1):13505. PMC: 8242026. DOI: 10.1038/s41598-021-92799-4. View

18.

Parsons D, Jones S, Zhang X, Cheng-Ho Lin J, Leary R, Angenendt P . An integrated genomic analysis of human glioblastoma multiforme. Science. 2008; 321(5897):1807-12. PMC: 2820389. DOI: 10.1126/science.1164382. View

19.

Karczewski K, Snyder M . Integrative omics for health and disease. Nat Rev Genet. 2018; 19(5):299-310. PMC: 5990367. DOI: 10.1038/nrg.2018.4. View

20.

Kvamme H, Borgan O . Continuous and discrete-time survival prediction with neural networks. Lifetime Data Anal. 2021; 27(4):710-736. PMC: 8536596. DOI: 10.1007/s10985-021-09532-6. View