Validation of ChatGPT 3.5 As a Tool to Optimize Readability of Patient-facing Craniofacial Education Materials

Overview

Journal Plast Reconstr Surg Glob Open

Specialty General Surgery

Date 2024 Feb 5

PMID 38313589

Authors

Medha Vallurupalli

Nikhil D Shah

Raj M Vyas

Affiliations

Soon will be listed here.

Abstract

Background: To address patient health literacy, the American Medical Association recommends that readability of patient education materials should not exceed a sixth grade reading level; the National Institutes of Health recommend no greater than an eigth-grade reading level. However, patient-facing materials in plastic surgery often remain at an above-recommended average reading level. The purpose of this study was to evaluate ChatGPT 3.5 as a tool for optimizing patient-facing craniofacial education materials.

Methods: Eighteen patient-facing craniofacial education materials were evaluated for readability by a traditional calculator and ChatGPT 3.5. The resulting scores were compared. The original excerpts were then inputted to ChatGPT 3.5 and simplified by the artificial intelligence tool. The simplified excerpts were scored by the calculators.

Results: The difference in scores for the original excerpts between the online calculator and ChatGPT 3.5 were not significant ( = 0.441). Additionally, the simplified excerpts' scores were significantly lower than the originals ( < 0.001), and the mean of the simplified excerpts was 7.78, less than the maximum recommended 8.

Conclusions: The use of ChatGPT 3.5 for simplification and readability analysis of patient-facing craniofacial materials is efficient and may help facilitate the conveyance of important health information. ChatGPT 3.5 rendered readability scores comparable to traditional readability calculators, in addition to excerpt-specific feedback. It was also able to simplify materials to the recommended grade levels. With human oversight, we validate this tool for readability analysis and simplification.

Citing Articles

Large language models in patient education: a scoping review of applications in medicine.

Aydin S, Karabacak M, Vlachos V, Margetis K Front Med (Lausanne). 2024; 11:1477898.

PMID: 39534227 PMC: 11554522. DOI: 10.3389/fmed.2024.1477898.

Large Language Models for Intraoperative Decision Support in Plastic Surgery: A Comparison between ChatGPT-4 and Gemini.

Gomez-Cabello C, Borna S, Pressman S, Haider S, Forte A Medicina (Kaunas). 2024; 60(6).

PMID: 38929573 PMC: 11205293. DOI: 10.3390/medicina60060957.

Artificial Intelligence in Postoperative Care: Assessing Large Language Models for Patient Recommendations in Plastic Surgery.

Gomez-Cabello C, Borna S, Pressman S, Haider S, Sehgal A, Leibovich B Healthcare (Basel). 2024; 12(11).

PMID: 38891158 PMC: 11171524. DOI: 10.3390/healthcare12111083.

Analysis of the Most Popular Online Ankle Fracture-Related Patient Education Materials.

Arif H, LeBrun G, Moore S, Friscia D Foot Ankle Orthop. 2024; 9(2):24730114241241310.

PMID: 38577700 PMC: 10989055. DOI: 10.1177/24730114241241310.

References

Seth A, Vargas C, Chuang D, Lee B . Readability Assessment of Patient Information about Lymphedema and Its Treatment. Plast Reconstr Surg. 2016; 137(2):287e-295e. DOI: 10.1097/01.prs.0000475747.95096.ab. View

Para A, Thelmo F, Rynecki N, Zelman B, Gupta R, Coban D . Evaluating the Readability of Online Patient Education Materials Related to Orthopedic Oncology. Orthopedics. 2020; 44(1):38-42. DOI: 10.3928/01477447-20201012-04. View

Vargas C, Ricci J, Lee M, Tobias A, Medalie D, Lee B . The accessibility, readability, and quality of online resources for gender affirming surgery. J Surg Res. 2017; 217:198-206. DOI: 10.1016/j.jss.2017.05.026. View

Patel A, Joshi C, Varghese J, Hassan A, Janis J, Galiano R . Do Websites Serve Our Patients Well? A Comparative Analysis of Online Information on Cosmetic Injectables. Plast Reconstr Surg. 2022; 149(4):655e-668e. DOI: 10.1097/PRS.0000000000008921. View

Greene S, Tuzzio L, Cherkin D . A framework for making patient-centered care front and center. Perm J. 2012; 16(3):49-53. PMC: 3442762. DOI: 10.7812/TPP/12-025. View

Theiss L, Wood T, McLeod M, Shao C, Dos Santos Marques I, Bajpai S . The association of health literacy and postoperative complications after colorectal surgery: A cohort study. Am J Surg. 2021; 223(6):1047-1052. PMC: 9012791. DOI: 10.1016/j.amjsurg.2021.10.024. View

Coughlin S, Vernon M, Hatzigeorgiou C, George V . Health Literacy, Social Determinants of Health, and Disease Prevention and Control. J Environ Health Sci. 2021; 6(1). PMC: 7889072. View

Rooney M, Santiago G, Perni S, Horowitz D, McCall A, Einstein A . Readability of Patient Education Materials From High-Impact Medical Journals: A 20-Year Analysis. J Patient Exp. 2021; 8:2374373521998847. PMC: 8205335. DOI: 10.1177/2374373521998847. View

Seo J, Goodman M, Politi M, Blanchard M, Kaphingst K . Effect of Health Literacy on Decision-Making Preferences among Medically Underserved Patients. Med Decis Making. 2016; 36(4):550-6. PMC: 5546799. DOI: 10.1177/0272989X16632197. View

10.

Eltorai A, Ghanian S, Adams Jr C, Born C, Daniels A . Readability of patient education materials on the american association for surgery of trauma website. Arch Trauma Res. 2014; 3(2):e18161. PMC: 4139691. DOI: 10.5812/atr.18161. View

11.

Barton N, Janis J . Missing the Mark: The State of Health Care Literacy in Plastic Surgery. Plast Reconstr Surg Glob Open. 2020; 8(5):e2856. PMC: 7572040. DOI: 10.1097/GOX.0000000000002856. View

12.

Hickey K, Masterson Creber R, Reading M, Sciacca R, Riga T, Frulla A . Low health literacy: Implications for managing cardiac patients in practice. Nurse Pract. 2018; 43(8):49-55. PMC: 6391993. DOI: 10.1097/01.NPR.0000541468.54290.49. View

13.

Ricci J, Vargas C, Chuang D, Lin S, Lee B . Readability assessment of online patient resources for breast augmentation surgery. Plast Reconstr Surg. 2015; 135(6):1573-1579. DOI: 10.1097/PRS.0000000000001230. View

14.

Kiwanuka E, Mehrzad R, Prsic A, Kwan D . Online Patient Resources for Gender Affirmation Surgery: An Analysis of Readability. Ann Plast Surg. 2017; 79(4):329-333. DOI: 10.1097/SAP.0000000000001159. View

15.

Tiourin E, Barton N, Janis J . Health Literacy in Plastic Surgery: A Scoping Review. Plast Reconstr Surg Glob Open. 2022; 10(4):e4247. PMC: 9007188. DOI: 10.1097/GOX.0000000000004247. View

16.

Chen A, Ruan Q, Bucknor A, Chattha A, Bletsis P, Furnas H . Social Media: Is the Message Reaching the Plastic Surgery Audience?. Plast Reconstr Surg. 2019; 144(3):773-781. DOI: 10.1097/PRS.0000000000005988. View

17.

Fanning J, Okamoto L, Levine E, McGee S, Janis J . Content and Readability of Online Recommendations for Breast Implant Size Selection. Plast Reconstr Surg Glob Open. 2023; 11(1):e4787. PMC: 9872969. DOI: 10.1097/GOX.0000000000004787. View

18.

Tran B, Singh M, Singhal D, Rudd R, Lee B . Readability, complexity, and suitability of online resources for mastectomy and lumpectomy. J Surg Res. 2017; 212:214-221. DOI: 10.1016/j.jss.2017.01.012. View

19.

Duff L, Scarsbrook A, Ravikumar N, Frood R, van Praagh G, Mackie S . An Automated Method for Artifical Intelligence Assisted Diagnosis of Active Aortitis Using Radiomic Analysis of FDG PET-CT Images. Biomolecules. 2023; 13(2). PMC: 9953018. DOI: 10.3390/biom13020343. View

20.

Ebrahimian S, Digumarthy S, Bizzo B, Primak A, Zimmermann M, Tarbiah M . Artificial Intelligence has Similar Performance to Subjective Assessment of Emphysema Severity on Chest CT. Acad Radiol. 2021; 29(8):1189-1195. DOI: 10.1016/j.acra.2021.09.007. View