ChatGPT-4o Can Serve As the Second Rater for Data Extraction in Systematic Reviews

Overview

Journal PLoS One

Specialties General Medicine
Science

Date 2025 Jan 8

PMID 39774443

Authors

Mette Motzfeldt Jensen

Mathias Brix Danielsen

Johannes Riis

Karoline Assifuah Kristjansen

Stig Andersen

Yoshiro Okubo

Martin Gronbech Jorgensen

Affiliations

Soon will be listed here.

Abstract

Background: Systematic reviews provide clarity of a bulk of evidence and support the transfer of knowledge from clinical trials to guidelines. Yet, they are time-consuming. Artificial intelligence (AI), like ChatGPT-4o, may streamline processes of data extraction, but its efficacy requires validation.

Objective: This study aims to (1) evaluate the validity of ChatGPT-4o for data extraction compared to human reviewers, and (2) test the reproducibility of ChatGPT-4o's data extraction.

Methods: We conducted a comparative study using papers from an ongoing systematic review on exercise to reduce fall risk. Data extracted by ChatGPT-4o were compared to a reference standard: data extracted by two independent human reviewers. The validity was assessed by categorizing the extracted data into five categories ranging from completely correct to false data. Reproducibility was evaluated by comparing data extracted in two separate sessions using different ChatGPT-4o accounts.

Results: ChatGPT-4o extracted a total of 484 data points across 11 papers. The AI's data extraction was 92.4% accurate (95% CI: 89.5% to 94.5%) and produced false data in 5.2% of cases (95% CI: 3.4% to 7.4%). The reproducibility between the two sessions was high, with an overall agreement of 94.1%. Reproducibility decreased when information was not reported in the papers, with an agreement of 77.2%.

Conclusion: Validity and reproducibility of ChatGPT-4o was high for data extraction for systematic reviews. ChatGPT-4o was qualified as a second reviewer for systematic reviews and showed potential for future advancements when summarizing data.

References

Feinstein A, Cicchetti D . High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol. 1990; 43(6):543-9. DOI: 10.1016/0895-4356(90)90158-l. View

Buscemi N, Hartling L, Vandermeer B, Tjosvold L, Klassen T . Single data extraction generated more errors than double data extraction in systematic reviews. J Clin Epidemiol. 2006; 59(7):697-703. DOI: 10.1016/j.jclinepi.2005.11.010. View

van Dijk S, Brusse-Keizer M, Bucsan C, van der Palen J, Doggen C, Lenferink A . Artificial intelligence in systematic reviews: promising when appropriately used. BMJ Open. 2023; 13(7):e072254. PMC: 10335470. DOI: 10.1136/bmjopen-2023-072254. View

Qureshi R, Shaughnessy D, Gill K, Robinson K, Li T, Agai E . Are ChatGPT and large language models "the answer" to bringing us closer to systematic review automation?. Syst Rev. 2023; 12(1):72. PMC: 10148473. DOI: 10.1186/s13643-023-02243-z. View

Lapping K, Marsh D, Rosenbaum J, Swedberg E, Sternin J, Sternin M . The positive deviance approach: challenges and opportunities for the future. Food Nutr Bull. 2002; 23(4 Suppl):130-7. View

Wongpakaran N, Wongpakaran T, Wedding D, Gwet K . A comparison of Cohen's Kappa and Gwet's AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples. BMC Med Res Methodol. 2013; 13:61. PMC: 3643869. DOI: 10.1186/1471-2288-13-61. View

Alyasiri O, Salman A, Akhtom D, Salisu S . ChatGPT revisited: Using ChatGPT-4 for finding references and editing language in medical scientific articles. J Stomatol Oral Maxillofac Surg. 2024; 125(5S2):101842. DOI: 10.1016/j.jormas.2024.101842. View

Mahuli S, Rai A, Mahuli A, Kumar A . Application ChatGPT in conducting systematic reviews and meta-analyses. Br Dent J. 2023; 235(2):90-92. DOI: 10.1038/s41415-023-6132-y. View

Borah R, Brown A, Capers P, Kaiser K . Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open. 2017; 7(2):e012545. PMC: 5337708. DOI: 10.1136/bmjopen-2016-012545. View

10.

Blaizot A, Veettil S, Saidoung P, Moreno-Garcia C, Wiratunga N, Aceves-Martins M . Using artificial intelligence methods for systematic review in health sciences: A systematic review. Res Synth Methods. 2022; 13(3):353-362. DOI: 10.1002/jrsm.1553. View

11.

Ghosh M, Mukherjee S, Ganguly A, Basuchowdhuri P, Naskar S, Ganguly D . AlpaPICO: Extraction of PICO frames from clinical trial documents using LLMs. Methods. 2024; 226:78-88. DOI: 10.1016/j.ymeth.2024.04.005. View

12.

Dos Santos A, da Silva E, Couto L, Reis G, Belo V . The use of artificial intelligence for automating or semi-automating biomedical literature analyses: A scoping review. J Biomed Inform. 2023; 142:104389. DOI: 10.1016/j.jbi.2023.104389. View

13.

Tran V, Gartlehner G, Yaacoub S, Boutron I, Schwingshackl L, Stadelmaier J . Sensitivity and Specificity of Using GPT-3.5 Turbo Models for Title and Abstract Screening in Systematic Reviews and Meta-analyses. Ann Intern Med. 2024; 177(6):791-799. DOI: 10.7326/M23-3389. View

14.

Feng Y, Liang S, Zhang Y, Chen S, Wang Q, Huang T . Automated medical literature screening using artificial intelligence: a systematic review and meta-analysis. J Am Med Inform Assoc. 2022; 29(8):1425-1432. PMC: 9277646. DOI: 10.1093/jamia/ocac066. View

15.

Flanagin A, Pirracchio R, Khera R, Berkwits M, Hswen Y, Bibbins-Domingo K . Reporting Use of AI in Research and Scholarly Publication-JAMA Network Guidance. JAMA. 2024; 331(13):1096-1098. DOI: 10.1001/jama.2024.3471. View