Performance of ChatGPT on USMLE: Potential for AI-assisted Medical Education Using Large Language Models

Overview

Journal PLOS Digit Health

Date 2023 Feb 22

PMID 36812645

Authors

Tiffany H Kung

Morgan Cheatham

Arielle Medenilla

Czarina Sillos

Lorie De Leon

Camille Elepano

Maria Madriaga

Rimel Aggabao

Giezel Diaz-Candido

James Maningo

Victor Tseng

Affiliations

Soon will be listed here.

Abstract

We evaluated the performance of a large language model called ChatGPT on the United States Medical Licensing Exam (USMLE), which consists of three exams: Step 1, Step 2CK, and Step 3. ChatGPT performed at or near the passing threshold for all three exams without any specialized training or reinforcement. Additionally, ChatGPT demonstrated a high level of concordance and insight in its explanations. These results suggest that large language models may have the potential to assist with medical education, and potentially, clinical decision-making.

Citing Articles

Comparing ChatGPT 4.0's Performance in Interpreting Thyroid Nodule Ultrasound Reports Using ACR-TI-RADS 2017: Analysis Across Different Levels of Ultrasound User Experience.

Wakonig K, Barisch S, Kozarzewski L, Dommerich S, Lerchbaumer M Diagnostics (Basel). 2025; 15(5).

PMID: 40075883 PMC: 11899695. DOI: 10.3390/diagnostics15050635.

Investigating the Accuracy and Consistency of ChatGPT in the Management of Achilles Tendon Ruptures.

Knee C, Campbell R, Sivakumar B, Wines A, Symes M Cureus. 2025; 17(2):e78433.

PMID: 40046346 PMC: 11882158. DOI: 10.7759/cureus.78433.

ChatGPT's Performance on Portuguese Medical Examination Questions: Comparative Analysis of ChatGPT-3.5 Turbo and ChatGPT-4o Mini.

Prazeres F JMIR Med Educ. 2025; 11:e65108.

PMID: 40043219 PMC: 11902880. DOI: 10.2196/65108.

GPT-4 generated answer rationales to multiple choice assessment questions in undergraduate medical education.

Chen P, Day W, Pekson R, Barrientos J, Burton W, Ludwig A BMC Med Educ. 2025; 25(1):333.

PMID: 40038669 PMC: 11877964. DOI: 10.1186/s12909-025-06862-z.

Evaluating base and retrieval augmented LLMs with document or online support for evidence based neurology.

Masanneck L, Meuth S, Pawlitzki M NPJ Digit Med. 2025; 8(1):137.

PMID: 40038423 PMC: 11880332. DOI: 10.1038/s41746-025-01536-y.

References

Delahanty R, Kaufman D, Jones S . Development and Evaluation of an Automated Machine Learning Algorithm for In-Hospital Mortality Risk Adjustment Among Critical Care Patients. Crit Care Med. 2018; 46(6):e481-e488. DOI: 10.1097/CCM.0000000000003011. View

Prasad V, Vandross A, Toomey C, Cheung M, Rho J, Quinn S . A decade of reversal: an analysis of 146 contradicted medical practices. Mayo Clin Proc. 2013; 88(8):790-8. DOI: 10.1016/j.mayocp.2013.05.012. View

McDermott M, Wang S, Marinsek N, Ranganath R, Foschini L, Ghassemi M . Reproducibility in machine learning for health research: Still a ways to go. Sci Transl Med. 2021; 13(586). DOI: 10.1126/scitranslmed.abb1655. View

Milne-Ives M, de Cock C, Lim E, Harper Shehadeh M, de Pennington N, Mole G . The Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review. J Med Internet Res. 2020; 22(10):e20346. PMC: 7644372. DOI: 10.2196/20346. View

Lievin V, Hother C, Motzfeldt A, Winther O . Can large language models reason about medical questions?. Patterns (N Y). 2024; 5(3):100943. PMC: 10935498. DOI: 10.1016/j.patter.2024.100943. View

Liu Y, Jain A, Eng C, Way D, Lee K, Bui P . A deep learning system for differential diagnosis of skin diseases. Nat Med. 2020; 26(6):900-908. DOI: 10.1038/s41591-020-0842-3. View

Plana D, Shung D, Grimshaw A, Saraf A, Sung J, Kann B . Randomized Clinical Trials of Machine Learning Interventions in Health Care: A Systematic Review. JAMA Netw Open. 2022; 5(9):e2233946. PMC: 9523495. DOI: 10.1001/jamanetworkopen.2022.33946. View

Abou-Hanna J, Owens S, Kinnucan J, Mian S, Kolars J . Resuscitating the Socratic Method: Student and Faculty Perspectives on Posing Probing Questions During Clinical Teaching. Acad Med. 2021; 96(1):113-117. DOI: 10.1097/ACM.0000000000003580. View

Chen P, Liu Y, Peng L . How to develop machine learning models for healthcare. Nat Mater. 2019; 18(5):410-414. DOI: 10.1038/s41563-019-0345-0. View

10.

Burk-Rafel J, Santen S, Purkiss J . Study Behaviors and USMLE Step 1 Performance: Implications of a Student Self-Directed Parallel Curriculum. Acad Med. 2017; 92:S67-S74. DOI: 10.1097/ACM.0000000000001916. View

11.

Nagpal K, Foote D, Liu Y, Chen P, Wulczyn E, Tan F . Development and validation of a deep learning algorithm for improving Gleason scoring of prostate cancer. NPJ Digit Med. 2019; 2:48. PMC: 6555810. DOI: 10.1038/s41746-019-0112-2. View

12.

Herrera-Perez D, Haslam A, Crain T, Gill J, Livingston C, Kaestner V . A comprehensive review of randomized clinical trials in three medical journals reveals 396 medical reversals. Elife. 2019; 8. PMC: 6559784. DOI: 10.7554/eLife.45183. View

13.

Gulshan V, Peng L, Coram M, Stumpe M, Wu D, Narayanaswamy A . Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. JAMA. 2016; 316(22):2402-2410. DOI: 10.1001/jama.2016.17216. View

14.

Vasey B, Nagendran M, Campbell B, Clifton D, Collins G, Denaxas S . Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. Nat Med. 2022; 28(5):924-933. DOI: 10.1038/s41591-022-01772-9. View

15.

Kan H, Kharrazi H, Chang H, Bodycombe D, Lemke K, Weiner J . Exploring the use of machine learning for risk adjustment: A comparison of standard and penalized linear regression models in predicting health care costs in older adults. PLoS One. 2019; 14(3):e0213258. PMC: 6402678. DOI: 10.1371/journal.pone.0213258. View

16.

Bala S, Keniston A, Burden M . Patient Perception of Plain-Language Medical Notes Generated Using Artificial Intelligence Software: Pilot Mixed-Methods Study. JMIR Form Res. 2020; 4(6):e16670. PMC: 7305564. DOI: 10.2196/16670. View

17.

Densen P . Challenges and opportunities facing medical education. Trans Am Clin Climatol Assoc. 2011; 122:48-58. PMC: 3116346. View

18.

Garcia-Vidal C, Sanjuan G, Puerta-Alcalde P, Moreno-Garcia E, Soriano A . Artificial intelligence to support clinical decision-making processes. EBioMedicine. 2019; 46:27-29. PMC: 6710912. DOI: 10.1016/j.ebiom.2019.07.019. View