ChatGPT's Performance in Cardiac Arrest and Bradycardia Simulations Using the American Heart Association's Advanced Cardiovascular Life Support Guidelines: Exploratory Study

Overview

Journal J Med Internet Res

Publisher JMIR Publications

Specialty Medical Informatics

Date 2024 Apr 22

PMID 38648098

Authors

Cecilia Pham

Romi Govender

Salik Tehami

Summer Chavez

Omolola E Adepoju

Winston Liaw

Affiliations

Soon will be listed here.

Abstract

Background: ChatGPT is the most advanced large language model to date, with prior iterations having passed medical licensing examinations, providing clinical decision support, and improved diagnostics. Although limited, past studies of ChatGPT's performance found that artificial intelligence could pass the American Heart Association's advanced cardiovascular life support (ACLS) examinations with modifications. ChatGPT's accuracy has not been studied in more complex clinical scenarios. As heart disease and cardiac arrest remain leading causes of morbidity and mortality in the United States, finding technologies that help increase adherence to ACLS algorithms, which improves survival outcomes, is critical.

Objective: This study aims to examine the accuracy of ChatGPT in following ACLS guidelines for bradycardia and cardiac arrest.

Methods: We evaluated the accuracy of ChatGPT's responses to 2 simulations based on the 2020 American Heart Association ACLS guidelines with 3 primary outcomes of interest: the mean individual step accuracy, the accuracy score per simulation attempt, and the accuracy score for each algorithm. For each simulation step, ChatGPT was scored for correctness (1 point) or incorrectness (0 points). Each simulation was conducted 20 times.

Results: ChatGPT's median accuracy for each step was 85% (IQR 40%-100%) for cardiac arrest and 30% (IQR 13%-81%) for bradycardia. ChatGPT's median accuracy over 20 simulation attempts for cardiac arrest was 69% (IQR 67%-74%) and for bradycardia was 42% (IQR 33%-50%). We found that ChatGPT's outputs varied despite consistent input, the same actions were persistently missed, repetitive overemphasis hindered guidance, and erroneous medication information was presented.

Conclusions: This study highlights the need for consistent and reliable guidance to prevent potential medical errors and optimize the application of ChatGPT to enhance its reliability and effectiveness in clinical practice.

Citing Articles

Qwen-2.5 Outperforms Other Large Language Models in the Chinese National Nursing Licensing Examination: Retrospective Cross-Sectional Comparative Study.

Zhu S, Hu W, Yang Z, Yan J, Zhang F JMIR Med Inform. 2025; 13:e63731.

PMID: 39793017 PMC: 11759905. DOI: 10.2196/63731.

AI in Dental Radiology-Improving the Efficiency of Reporting With ChatGPT: Comparative Study.

Stephan D, Bertsch A, Burwinkel M, Vinayahalingam S, Al-Nawas B, Kammerer P J Med Internet Res. 2024; 26():e60684.

PMID: 39714078 PMC: 11704643. DOI: 10.2196/60684.

Harnessing the Power of ChatGPT in Cardiovascular Medicine: Innovations, Challenges, and Future Directions.

Leon M, Ruaengsri C, Pelletier G, Bethencourt D, Shibata M, Flores M J Clin Med. 2024; 13(21).

PMID: 39518681 PMC: 11546989. DOI: 10.3390/jcm13216543.

Evaluation of Mortality Rate Disparities for Cardiac Arrest Between Urban and Rural Cohorts in the United States Using the Centers for Disease Control and Prevention Wide-Ranging Online Data for Epidemiologic Research (CDC WONDER) Database.

Arefin S, Alluri A, Barua M, Patel T, Kandhalu S Cureus. 2024; 16(9):e68803.

PMID: 39376888 PMC: 11456408. DOI: 10.7759/cureus.68803.

References

Ahn C . Exploring ChatGPT for information of cardiopulmonary resuscitation. Resuscitation. 2023; 185:109729. DOI: 10.1016/j.resuscitation.2023.109729. View

Hunt E, Vera K, Diener-West M, Haggerty J, Nelson K, Shaffner D . Delays and errors in cardiopulmonary resuscitation and defibrillation by pediatric residents during simulated cardiopulmonary arrests. Resuscitation. 2009; 80(7):819-25. DOI: 10.1016/j.resuscitation.2009.03.020. View

LeCun Y, Bengio Y, Hinton G . Deep learning. Nature. 2015; 521(7553):436-44. DOI: 10.1038/nature14539. View

Honarmand K, Mepham C, Ainsworth C, Khalid Z . Adherence to advanced cardiovascular life support (ACLS) guidelines during in-hospital cardiac arrest is associated with improved outcomes. Resuscitation. 2018; 129:76-81. DOI: 10.1016/j.resuscitation.2018.06.005. View

Hejjaji V, Malik A, Peri-Okonny P, Thomas M, Tang Y, Wooldridge D . Mobile App to Improve House Officers' Adherence to Advanced Cardiac Life Support Guidelines: Quality Improvement Study. JMIR Mhealth Uhealth. 2020; 8(5):e15762. PMC: 7267993. DOI: 10.2196/15762. View

Siebert J, Ehrler F, Gervaix A, Haddad K, Lacroix L, Schrurs P . Adherence to AHA Guidelines When Adapted for Augmented Reality Glasses for Assisted Pediatric Cardiopulmonary Resuscitation: A Randomized Controlled Trial. J Med Internet Res. 2017; 19(5):e183. PMC: 5468544. DOI: 10.2196/jmir.7379. View

Siebert J, Lacroix L, Cantais A, Manzano S, Ehrler F . The Impact of a Tablet App on Adherence to American Heart Association Guidelines During Simulated Pediatric Cardiopulmonary Resuscitation: Randomized Controlled Trial. J Med Internet Res. 2020; 22(5):e17792. PMC: 7287744. DOI: 10.2196/17792. View

Gilson A, Safranek C, Huang T, Socrates V, Chi L, Taylor R . How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment. JMIR Med Educ. 2023; 9:e45312. PMC: 9947764. DOI: 10.2196/45312. View

Crowley C, Salciccioli J, Kim E . The association between ACLS guideline deviations and outcomes from in-hospital cardiac arrest. Resuscitation. 2020; 153:65-70. PMC: 7750980. DOI: 10.1016/j.resuscitation.2020.05.042. View

10.

Corazza F, Fiorese E, Arpone M, Tardini G, Frigo A, Cheng A . The impact of cognitive aids on resuscitation performance in in-hospital cardiac arrest scenarios: a systematic review and meta-analysis. Intern Emerg Med. 2022; 17(7):2143-2158. PMC: 9420676. DOI: 10.1007/s11739-022-03041-6. View

11.

Zhu L, Mou W, Yang T, Chen R . ChatGPT can pass the AHA exams: Open-ended questions outperform multiple-choice format. Resuscitation. 2023; 188:109783. DOI: 10.1016/j.resuscitation.2023.109783. View

12.

Sanders A, Berg R, Burress M, Genova R, Kern K, Ewy G . The efficacy of an ACLS training program for resuscitation from cardiac arrest in a rural community. Ann Emerg Med. 1994; 23(1):56-9. DOI: 10.1016/s0196-0644(94)70009-5. View

13.

Brin D, Sorin V, Vaid A, Soroush A, Glicksberg B, Charney A . Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments. Sci Rep. 2023; 13(1):16492. PMC: 10543445. DOI: 10.1038/s41598-023-43436-9. View

14.

Kung T, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepano C . Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digit Health. 2023; 2(2):e0000198. PMC: 9931230. DOI: 10.1371/journal.pdig.0000198. View

15.

Sodhi K, Singla M, Shrivastava A . Impact of advanced cardiac life support training program on the outcome of cardiopulmonary resuscitation in a tertiary care hospital. Indian J Crit Care Med. 2012; 15(4):209-12. PMC: 3271556. DOI: 10.4103/0972-5229.92070. View

16.

Liu S, Wright A, Patterson B, Wanderer J, Turer R, Nelson S . Using AI-generated suggestions from ChatGPT to optimize clinical decision support. J Am Med Inform Assoc. 2023; 30(7):1237-1245. PMC: 10280357. DOI: 10.1093/jamia/ocad072. View

17.

Fijacko N, Gosak L, Stiglic G, Picard C, John Douma M . Can ChatGPT pass the life support exams without entering the American heart association course?. Resuscitation. 2023; 185:109732. DOI: 10.1016/j.resuscitation.2023.109732. View

18.

Brophy S, McCue M, Reel R, Jones T, Dias R . The impact of a smartphone-based cognitive aid on clinical performance during cardiac arrest simulations: A randomized controlled trial. AEM Educ Train. 2023; 7(3):e10880. PMC: 10245296. DOI: 10.1002/aet2.10880. View

19.

Corazza F, Arpone M, Tardini G, Stritoni V, Mormando G, Graziano A . Effectiveness of a Novel Tablet Application in Reducing Guideline Deviations During Pediatric Cardiac Arrest: A Randomized Clinical Trial. JAMA Netw Open. 2023; 6(8):e2327272. PMC: 10401301. DOI: 10.1001/jamanetworkopen.2023.27272. View

20.

Harskamp R, De Clercq L . Performance of ChatGPT as an AI-assisted decision support tool in medicine: a proof-of-concept study for interpreting symptoms and management of common cardiac conditions (AMSTELHEART-2). Acta Cardiol. 2024; 79(3):358-366. DOI: 10.1080/00015385.2024.2303528. View