Comparison of Chest Radiograph Captions Based on Natural Language Processing Vs Completed by Radiologists

Overview

Journal JAMA Netw Open

Publisher American Medical Association

Specialty General Medicine

Date 2023 Feb 8

PMID 36753278

Authors

Yaping Zhang

Mingqian Liu

Lu Zhang

Lingyun Wang

Keke Zhao

Shundong Hu

Xu Chen

Xueqian Xie

Affiliations

Soon will be listed here.

Abstract

Importance: Artificial intelligence (AI) can interpret abnormal signs in chest radiography (CXR) and generate captions, but a prospective study is needed to examine its practical value.

Objective: To prospectively compare natural language processing (NLP)-generated CXR captions and the diagnostic findings of radiologists.

Design, Setting, And Participants: A multicenter diagnostic study was conducted. The training data set included CXR images and reports retrospectively collected from February 1, 2014, to February 28, 2018. The retrospective test data set included consecutive images and reports from April 1 to July 31, 2019. The prospective test data set included consecutive images and reports from May 1 to September 30, 2021.

Exposures: A bidirectional encoder representation from a transformers model was used to extract language entities and relationships from unstructured CXR reports to establish 23 labels of abnormal signs to train convolutional neural networks. The participants in the prospective test group were randomly assigned to 1 of 3 different caption generation models: a normal template, NLP-generated captions, and rule-based captions based on convolutional neural networks. For each case, a resident drafted the report based on the randomly assigned captions and an experienced radiologist finalized the report blinded to the original captions. A total of 21 residents and 19 radiologists were involved.

Main Outcomes And Measures: Time to write reports based on different caption generation models.

Results: The training data set consisted of 74 082 cases (39 254 [53.0%] women; mean [SD] age, 50.0 [17.1] years). In the retrospective (n = 8126; 4345 [53.5%] women; mean [SD] age, 47.9 [15.9] years) and prospective (n = 5091; 2416 [47.5%] women; mean [SD] age, 45.1 [15.6] years) test data sets, the mean (SD) area under the curve of abnormal signs was 0.87 (0.11) in the retrospective data set and 0.84 (0.09) in the prospective data set. The residents' mean (SD) reporting time using the NLP-generated model was 283 (37) seconds-significantly shorter than the normal template (347 [58] seconds; P < .001) and the rule-based model (296 [46] seconds; P < .001). The NLP-generated captions showed the highest similarity to the final reports with a mean (SD) bilingual evaluation understudy score of 0.69 (0.24)-significantly higher than the normal template (0.37 [0.09]; P < .001) and the rule-based model (0.57 [0.19]; P < .001).

Conclusions And Relevance: In this diagnostic study of NLP-generated CXR captions, prior information provided by NLP was associated with greater efficiency in the reporting process, while maintaining good consistency with the findings of radiologists.

Citing Articles

A Transformer-Based Pipeline for German Clinical Document De-Identification.

Arzideh K, Baldini G, Winnekens P, Friedrich C, Nensa F, Idrissi-Yaghir A Appl Clin Inform. 2025; 16(1):31-43.

PMID: 39778706 PMC: 11710903. DOI: 10.1055/a-2424-1989.

Advances in research and application of artificial intelligence and radiomic predictive models based on intracranial aneurysm images.

Wen Z, Wang Y, Zhong Y, Hu Y, Yang C, Peng Y Front Neurol. 2024; 15:1391382.

PMID: 38694771 PMC: 11061371. DOI: 10.3389/fneur.2024.1391382.

Preliminary Evidence of the Use of Generative AI in Health Care Clinical Services: Systematic Narrative Review.

Yim D, Khuntia J, Parameswaran V, Meyers A JMIR Med Inform. 2024; 12:e52073.

PMID: 38506918 PMC: 10993141. DOI: 10.2196/52073.

Fully automated artificial intelligence-based coronary CT angiography image processing: efficiency, diagnostic capability, and risk stratification.

Zhang Y, Feng Y, Sun J, Zhang L, Ding Z, Wang L Eur Radiol. 2024; 34(8):4909-4919.

PMID: 38193925 DOI: 10.1007/s00330-023-10494-6.

References

Jang S, Song H, Shin Y, Kim J, Kim J, Lee K . Deep Learning-based Automatic Detection Algorithm for Reducing Overlooked Lung Cancers on Chest Radiographs. Radiology. 2020; 296(3):652-661. DOI: 10.1148/radiol.2020200165. View

Iwamura K, Louhi Kasahara J, Moro A, Yamashita A, Asama H . Image Captioning Using Motion-CNN with Object Detection. Sensors (Basel). 2021; 21(4). PMC: 7916682. DOI: 10.3390/s21041270. View

Zhang L, Xu Z, Jiang B, Zhang Y, Wang L, de Bock G . Machine-learning-based radiomics identifies atrial fibrillation on the epicardial fat in contrast-enhanced and non-enhanced chest CT. Br J Radiol. 2022; 95(1135):20211274. PMC: 10996326. DOI: 10.1259/bjr.20211274. View

Singh R, Kalra M, Nitiwarangkul C, Patti J, Homayounieh F, Padole A . Deep learning in chest radiography: Detection of findings and presence of change. PLoS One. 2018; 13(10):e0204155. PMC: 6171827. DOI: 10.1371/journal.pone.0204155. View

Pesce E, Withey S, Ypsilantis P, Bakewell R, Goh V, Montana G . Learning to detect chest radiographs containing pulmonary lesions using visual attention networks. Med Image Anal. 2019; 53:26-38. DOI: 10.1016/j.media.2018.12.007. View

Park S . Diagnostic Case-Control versus Diagnostic Cohort Studies for Clinical Validation of Artificial Intelligence Algorithm Performance. Radiology. 2018; 290(1):272-273. DOI: 10.1148/radiol.2018182294. View

Wu J, Wong K, Gur Y, Ansari N, Karargyris A, Sharma A . Comparison of Chest Radiograph Interpretations by Artificial Intelligence Algorithm vs Radiology Residents. JAMA Netw Open. 2020; 3(10):e2022779. PMC: 7547369. DOI: 10.1001/jamanetworkopen.2020.22779. View

Donald J, Barnard S . Common patterns in 558 diagnostic radiology errors. J Med Imaging Radiat Oncol. 2012; 56(2):173-8. DOI: 10.1111/j.1754-9485.2012.02348.x. View

Oakden-Rayner L . Exploring Large-scale Public Medical Image Datasets. Acad Radiol. 2019; 27(1):106-112. DOI: 10.1016/j.acra.2019.10.006. View

10.

Nam J, Park S, Hwang E, Lee J, Jin K, Lim K . Development and Validation of Deep Learning-based Automatic Detection Algorithm for Malignant Pulmonary Nodules on Chest Radiographs. Radiology. 2018; 290(1):218-228. DOI: 10.1148/radiol.2018180237. View

11.

Elkin P, Froehling D, Wahner-Roedler D, Trusko B, Welsh G, Ma H . NLP-based identification of pneumonia cases from free-text radiological reports. AMIA Annu Symp Proc. 2008; :172-6. PMC: 2656026. View

12.

Jiang B, Zhang Y, Zhang L, de Bock G, Vliegenthart R, Xie X . Human-recognizable CT image features of subsolid lung nodules associated with diagnosis and classification by convolutional neural networks. Eur Radiol. 2021; 31(10):7303-7315. DOI: 10.1007/s00330-021-07901-1. View

13.

Wolterink J, Leiner T, Viergever M, Isgum I . Generative Adversarial Networks for Noise Reduction in Low-Dose CT. IEEE Trans Med Imaging. 2017; 36(12):2536-2545. DOI: 10.1109/TMI.2017.2708987. View

14.

Shah S, Taju S, Ho Q, Nguyen T, Ou Y . GT-Finder: Classify the family of glucose transporters with pre-trained BERT language models. Comput Biol Med. 2021; 131:104259. DOI: 10.1016/j.compbiomed.2021.104259. View

15.

Liu Z, Chen Y, Tang B, Wang X, Chen Q, Li H . Automatic de-identification of electronic medical records using token-level and character-level conditional random fields. J Biomed Inform. 2015; 58 Suppl:S47-S52. PMC: 4988843. DOI: 10.1016/j.jbi.2015.06.009. View

16.

Homayounieh F, Digumarthy S, Ebrahimian S, Rueckel J, Hoppe B, Sabel B . An Artificial Intelligence-Based Chest X-ray Model on Human Nodule Detection Accuracy From a Multicenter Study. JAMA Netw Open. 2021; 4(12):e2141096. PMC: 8717119. DOI: 10.1001/jamanetworkopen.2021.41096. View

17.

Rueckel J, Huemmer C, Fieselmann A, Ghesu F, Mansoor A, Schachtner B . Pneumothorax detection in chest radiographs: optimizing artificial intelligence system for accuracy and confounding bias reduction using in-image annotations in algorithm training. Eur Radiol. 2021; 31(10):7888-7900. PMC: 8452588. DOI: 10.1007/s00330-021-07833-w. View

18.

Harvey H, Gilman M, Wu C, Cushing M, Halpern E, Zhao J . Diagnostic yield of recommendations for chest CT examination prompted by outpatient chest radiographic findings. Radiology. 2014; 275(1):262-71. PMC: 4455667. DOI: 10.1148/radiol.14140583. View

19.

Zhang Y, Liu M, Hu S, Shen Y, Lan J, Jiang B . Development and multicenter validation of chest X-ray radiography interpretations based on natural language processing. Commun Med (Lond). 2022; 1:43. PMC: 9053275. DOI: 10.1038/s43856-021-00043-x. View

20.

Singh V, Danda V, Gorniak R, Flanders A, Lakhani P . Assessment of Critical Feeding Tube Malpositions on Radiographs Using Deep Learning. J Digit Imaging. 2019; 32(4):651-655. PMC: 6646608. DOI: 10.1007/s10278-019-00229-9. View