Reliability of Medical Information Provided by ChatGPT: Assessment Against Clinical Guidelines and Patient Information Quality Instrument

Overview

Journal J Med Internet Res

Publisher JMIR Publications

Specialty Medical Informatics

Date 2023 Jun 30

PMID 37389908

Authors

Harriet Louise Walker

Shahi Ghani

Christoph Kuemmerli

Christian Andreas Nebiker

Beat Peter Muller

Dimitri Aristotle Raptis

Sebastian Manuel Staubli

Affiliations

Soon will be listed here.

Abstract

Background: ChatGPT-4 is the latest release of a novel artificial intelligence (AI) chatbot able to answer freely formulated and complex questions. In the near future, ChatGPT could become the new standard for health care professionals and patients to access medical information. However, little is known about the quality of medical information provided by the AI.

Objective: We aimed to assess the reliability of medical information provided by ChatGPT.

Methods: Medical information provided by ChatGPT-4 on the 5 hepato-pancreatico-biliary (HPB) conditions with the highest global disease burden was measured with the Ensuring Quality Information for Patients (EQIP) tool. The EQIP tool is used to measure the quality of internet-available information and consists of 36 items that are divided into 3 subsections. In addition, 5 guideline recommendations per analyzed condition were rephrased as questions and input to ChatGPT, and agreement between the guidelines and the AI answer was measured by 2 authors independently. All queries were repeated 3 times to measure the internal consistency of ChatGPT.

Results: Five conditions were identified (gallstone disease, pancreatitis, liver cirrhosis, pancreatic cancer, and hepatocellular carcinoma). The median EQIP score across all conditions was 16 (IQR 14.5-18) for the total of 36 items. Divided by subsection, median scores for content, identification, and structure data were 10 (IQR 9.5-12.5), 1 (IQR 1-1), and 4 (IQR 4-5), respectively. Agreement between guideline recommendations and answers provided by ChatGPT was 60% (15/25). Interrater agreement as measured by the Fleiss κ was 0.78 (P<.001), indicating substantial agreement. Internal consistency of the answers provided by ChatGPT was 100%.

Conclusions: ChatGPT provides medical information of comparable quality to available static internet information. Although currently of limited quality, large language models could become the future standard for patients and health care professionals to gather medical information.

Citing Articles

Revolutionizing MASLD: How Artificial Intelligence Is Shaping the Future of Liver Care.

Pugliese N, Bertazzoni A, Hassan C, Schattenberg J, Aghemo A Cancers (Basel). 2025; 17(5).

PMID: 40075570 PMC: 11899536. DOI: 10.3390/cancers17050722.

Artificial intelligence in healthcare education: evaluating the accuracy of ChatGPT, Copilot, and Google Gemini in cardiovascular pharmacology.

Salman I, Ameer O, Khanfar M, Hsieh Y Front Med (Lausanne). 2025; 12:1495378.

PMID: 40046930 PMC: 11879995. DOI: 10.3389/fmed.2025.1495378.

Comparing the performance of a large language model and naive human interviewers in interviewing children about a witnessed mock-event.

Sun Y, Pang H, Jarvilehto L, Zhang O, Shapiro D, Korkman J PLoS One. 2025; 20(2):e0316317.

PMID: 40019879 PMC: 11870376. DOI: 10.1371/journal.pone.0316317.

Artificial Intelligence (AI) - Powered Documentation Systems in Healthcare: A Systematic Review.

Bracken A, Reilly C, Feeley A, Sheehan E, Merghani K, Feeley I J Med Syst. 2025; 49(1):28.

PMID: 39966286 PMC: 11835907. DOI: 10.1007/s10916-025-02157-4.

Exploring the Utility of ChatGPT in Cleft Lip Repair Education.

Mahedia M, Rohrich R, Sadiq K, Bailey L, Harrison L, Hallac R J Clin Med. 2025; 14(3).

PMID: 39941663 PMC: 11818196. DOI: 10.3390/jcm14030993.

References

Stevens L, Guo M, Brown Z, Ejaz A, Pawlik T, Cloyd J . Evaluating the Quality of Online Information Regarding Neoadjuvant Therapy for Pancreatic Cancer. J Gastrointest Cancer. 2022; 54(3):890-896. DOI: 10.1007/s12029-022-00879-z. View

Vetter D, Ruhwinkel H, Raptis D, Bueter M . Quality Assessment of Information on Bariatric Surgery Websites. Obes Surg. 2017; 28(5):1240-1247. DOI: 10.1007/s11695-017-2983-0. View

Salvagno M, Taccone F, Gerli A . Can artificial intelligence help for scientific writing?. Crit Care. 2023; 27(1):75. PMC: 9960412. DOI: 10.1186/s13054-023-04380-2. View

Tsytsarev V . Methodological aspects of studying the mechanisms of consciousness. Behav Brain Res. 2021; 419:113684. DOI: 10.1016/j.bbr.2021.113684. View

Liebrenz M, Schleifer R, Buadze A, Bhugra D, Smith A . Generating scholarly content with ChatGPT: ethical challenges for medical publishing. Lancet Digit Health. 2023; 5(3):e105-e106. DOI: 10.1016/S2589-7500(23)00019-5. View

Charvet-Berard A, Chopard P, Perneger T . Measuring quality of patient information documents with an expanded EQIP scale. Patient Educ Couns. 2008; 70(3):407-11. DOI: 10.1016/j.pec.2007.11.018. View

The Lancet Digital Health . ChatGPT: friend or foe?. Lancet Digit Health. 2023; 5(3):e102. DOI: 10.1016/S2589-7500(23)00023-7. View

van Dis E, Bollen J, Zuidema W, van Rooij R, Bockting C . ChatGPT: five priorities for research. Nature. 2023; 614(7947):224-226. DOI: 10.1038/d41586-023-00288-7. View

Kwan L, Yip H, Tan S, Fan K . A quality assessment of online patient information regarding tonsillitis using the EQIP tool. Int J Pediatr Otorhinolaryngol. 2022; 159:111224. DOI: 10.1016/j.ijporl.2022.111224. View

10.

. The global burden of cancer attributable to risk factors, 2010-19: a systematic analysis for the Global Burden of Disease Study 2019. Lancet. 2022; 400(10352):563-591. PMC: 9395583. DOI: 10.1016/S0140-6736(22)01438-6. View

11.

Springfeld C, Ferrone C, Katz M, Philip P, Hong T, Hackert T . Neoadjuvant therapy for pancreatic cancer. Nat Rev Clin Oncol. 2023; 20(5):318-337. DOI: 10.1038/s41571-023-00746-1. View

12.

Sebastian G, George A, Jackson Jr G . Persuading Patients Using Rhetoric to Improve Artificial Intelligence Adoption: Experimental Study. J Med Internet Res. 2023; 25:e41430. PMC: 10131865. DOI: 10.2196/41430. View

13.

Moult B, Franck L, Brady H . Ensuring quality information for patients: development and preliminary validation of a new instrument to improve the quality of written health care information. Health Expect. 2004; 7(2):165-75. PMC: 5060233. DOI: 10.1111/j.1369-7625.2004.00273.x. View

14.

Ghani S, Fan K, Fan K, Lenti L, Raptis D . Using the Ensuring Quality Information for Patients Tool to Assess Patient Information on Appendicitis Websites: Systematic Search and Evaluation. J Med Internet Res. 2021; 23(3):e22618. PMC: 8035662. DOI: 10.2196/22618. View

15.

Raptis D, Sinanyan M, Ghani S, Soggiu F, Gilliland J, Imber C . Quality assessment of patient information on the management of gallstone disease in the internet - A systematic analysis using the modified ensuring quality information for patients tool. HPB (Oxford). 2019; 21(12):1632-1640. DOI: 10.1016/j.hpb.2019.03.355. View

16.

Alkaissi H, McFarlane S . Artificial Hallucinations in ChatGPT: Implications in Scientific Writing. Cureus. 2023; 15(2):e35179. PMC: 9939079. DOI: 10.7759/cureus.35179. View

17.

Eysenbach G . The Role of ChatGPT, Generative Language Models, and Artificial Intelligence in Medical Education: A Conversation With ChatGPT and a Call for Papers. JMIR Med Educ. 2023; 9:e46885. PMC: 10028514. DOI: 10.2196/46885. View

18.

. EASL Clinical Practice Guidelines: Management of hepatocellular carcinoma. J Hepatol. 2018; 69(1):182-236. DOI: 10.1016/j.jhep.2018.03.019. View

19.

Fan K, Ghani S, Machairas N, Lenti L, Fan K, Richardson D . COVID-19 prevention and treatment information on the internet: a systematic analysis and quality assessment. BMJ Open. 2020; 10(9):e040487. PMC: 7485261. DOI: 10.1136/bmjopen-2020-040487. View

20.

Gilson A, Safranek C, Huang T, Socrates V, Chi L, Taylor R . How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment. JMIR Med Educ. 2023; 9:e45312. PMC: 9947764. DOI: 10.2196/45312. View