Medical Image Captioning Via Generative Pretrained Transformers

Overview

Journal Sci Rep

Specialty Science

Date 2023 Mar 14

PMID 36914733

Authors

Alexander Selivanov

Oleg Y Rogov

Daniil Chesakov

Artem Shelmanov

Irina Fedulova

Dmitry V Dylov

Affiliations

Soon will be listed here.

Abstract

The proposed model for automatic clinical image caption generation combines the analysis of radiological scans with structured patient information from the textual records. It uses two language models, the Show-Attend-Tell and the GPT-3, to generate comprehensive and descriptive radiology records. The generated textual summary contains essential information about pathologies found, their location, along with the 2D heatmaps that localize each pathology on the scans. The model has been tested on two medical datasets, the Open-I, MIMIC-CXR, and the general-purpose MS-COCO, and the results measured with natural language assessment metrics demonstrated its efficient applicability to chest X-ray image captioning.

Citing Articles

Towards a holistic framework for multimodal LLM in 3D brain CT radiology report generation.

Li C, Chang K, Yang C, Wu H, Chen W, Bansal H Nat Commun. 2025; 16(1):2258.

PMID: 40050277 PMC: 11885477. DOI: 10.1038/s41467-025-57426-0.

Transforming Healthcare: Artificial Intelligence (AI) Applications in Medical Imaging and Drug Response Prediction.

Prathaban K, Hande M Genome Integr. 2025; 15():e20240002.

PMID: 39845982 PMC: 11752870. DOI: 10.14293/genint.15.1.002.

Large Language Models in Biomedical and Health Informatics: A Review with Bibliometric Analysis.

Yu H, Fan L, Li L, Zhou J, Ma Z, Xian L J Healthc Inform Res. 2024; 8(4):658-711.

PMID: 39463859 PMC: 11499577. DOI: 10.1007/s41666-024-00171-8.

A dental intraoral image dataset of gingivitis for image captioning.

Duy H, Hue T, Son T, Nghia L, Lan L, Duc N Data Brief. 2024; 57:110960.

PMID: 39386321 PMC: 11462013. DOI: 10.1016/j.dib.2024.110960.

Zero-shot Learning with Minimum Instruction to Extract Social Determinants and Family History from Clinical Notes using GPT Model.

Bhate N, Mittal A, He Z, Luo X Proc IEEE Int Conf Big Data. 2024; 2023:1476-1480.

PMID: 39101057 PMC: 11295958. DOI: 10.1109/BigData59044.2023.10386811.

References

Demner-Fushman D, Kohli M, Rosenman M, Shooshan S, Rodriguez L, Antani S . Preparing a collection of radiology examinations for distribution and retrieval. J Am Med Inform Assoc. 2015; 23(2):304-10. PMC: 5009925. DOI: 10.1093/jamia/ocv080. View

Gurgitano M, Angileri S, Roda G, Liguori A, Pandolfi M, Ierardi A . Interventional Radiology ex-machina: impact of Artificial Intelligence on practice. Radiol Med. 2021; 126(7):998-1006. PMC: 8050998. DOI: 10.1007/s11547-021-01351-x. View

Sermesant M, Delingette H, Cochet H, Jais P, Ayache N . Applications of artificial intelligence in cardiovascular imaging. Nat Rev Cardiol. 2021; 18(8):600-609. DOI: 10.1038/s41569-021-00527-2. View

Bodenreider O . The Unified Medical Language System (UMLS): integrating biomedical terminology. Nucleic Acids Res. 2003; 32(Database issue):D267-70. PMC: 308795. DOI: 10.1093/nar/gkh061. View

Belikova K, Rogov O, Rybakov A, Maslov M, Dylov D . Deep negative volume segmentation. Sci Rep. 2021; 11(1):16292. PMC: 8357924. DOI: 10.1038/s41598-021-95526-1. View

Hochreiter S, Schmidhuber J . Long short-term memory. Neural Comput. 1997; 9(8):1735-80. DOI: 10.1162/neco.1997.9.8.1735. View

Kraljevic Z, Searle T, Shek A, Roguski L, Noor K, Bean D . Multi-domain clinical natural language processing with MedCAT: The Medical Concept Annotation Toolkit. Artif Intell Med. 2021; 117:102083. DOI: 10.1016/j.artmed.2021.102083. View

Donahue J, Hendricks L, Rohrbach M, Venugopalan S, Guadarrama S, Saenko K . Long-Term Recurrent Convolutional Networks for Visual Recognition and Description. IEEE Trans Pattern Anal Mach Intell. 2016; 39(4):677-691. DOI: 10.1109/TPAMI.2016.2599174. View

Karpathy A, Fei-Fei L . Deep Visual-Semantic Alignments for Generating Image Descriptions. IEEE Trans Pattern Anal Mach Intell. 2016; 39(4):664-676. DOI: 10.1109/TPAMI.2016.2598339. View

10.

Kung T, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepano C . Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digit Health. 2023; 2(2):e0000198. PMC: 9931230. DOI: 10.1371/journal.pdig.0000198. View

11.

Chan Y, Zeng Y, Wu H, Wu M, Sun H . Effective Pneumothorax Detection for Chest X-Ray Images Using Local Binary Pattern and Support Vector Machine. J Healthc Eng. 2018; 2018:2908517. PMC: 5903299. DOI: 10.1155/2018/2908517. View

12.

Peng Y, Wang X, Lu L, Bagheri M, Summers R, Lu Z . NegBio: a high-performance tool for negation and uncertainty detection in radiology reports. AMIA Jt Summits Transl Sci Proc. 2018; 2017:188-196. PMC: 5961822. View

13.

Johnson A, Pollard T, Berkowitz S, Greenbaum N, Lungren M, Deng C . MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Sci Data. 2019; 6(1):317. PMC: 6908718. DOI: 10.1038/s41597-019-0322-0. View

14.

Monshi M, Poon J, Chung V . Deep learning in generating radiology reports: A survey. Artif Intell Med. 2020; 106:101878. PMC: 7227610. DOI: 10.1016/j.artmed.2020.101878. View