Automated Detection of Adverse Events Using Natural Language Processing of Discharge Summaries

Overview

Journal J Am Med Inform Assoc

Publisher Oxford University Press

Specialty Medical Informatics

Date 2005 Apr 2

PMID 15802475

Citations 116

Authors

Genevieve B Melton

George Hripcsak

Affiliations

Soon will be listed here.

Abstract

Objective: To determine whether natural language processing (NLP) can effectively detect adverse events defined in the New York Patient Occurrence Reporting and Tracking System (NYPORTS) using discharge summaries.

Design: An adverse event detection system for discharge summaries using the NLP system MedLEE was constructed to identify 45 NYPORTS event types. The system was first applied to a random sample of 1,000 manually reviewed charts. The system then processed all inpatient cases with electronic discharge summaries for two years. All system-identified events were reviewed, and performance was compared with traditional reporting.

Measurements: System sensitivity, specificity, and predictive value, with manual review serving as the gold standard.

Results: The system correctly identified 16 of 65 events in 1,000 charts. Of 57,452 total electronic discharge summaries, the system identified 1,590 events in 1,461 cases, and manual review verified 704 events in 652 cases, resulting in an overall sensitivity of 0.28 (95% confidence interval [CI]: 0.17-0.42), specificity of 0.985 (CI: 0.984-0.986), and positive predictive value of 0.45 (CI: 0.42-0.47) for detecting cases with events and an average specificity of 0.9996 (CI: 0.9996-0.9997) per event type. Traditional event reporting detected 322 events during the period (sensitivity 0.09), of which the system identified 110 as well as 594 additional events missed by traditional methods.

Conclusion: NLP is an effective technique for detecting a broad range of adverse events in text documents and outperformed traditional and previous automated adverse event detection methods.

Citing Articles

Real-World Evidence on Adverse Events and Healthcare Resource Utilization in Patients with Chronic Lymphocytic Leukaemia in Spain Using Natural Language Processing: The SRealCLL Study.

Abrisqueta-Costa P, Garcia-Marco J, Gutierrez A, Hernandez-Rivas J, Andreu-Lapiedra R, Arguello-Tomas M Cancers (Basel). 2024; 16(23).

PMID: 39682190 PMC: 11639754. DOI: 10.3390/cancers16234004.

Artificial intelligence in robot-assisted radical prostatectomy: where do we stand today?.

Carbin D, Shah A, Kusuma V J Robot Surg. 2024; 18(1):404.

PMID: 39527349 DOI: 10.1007/s11701-024-02143-x.

The Impact of Artificial Intelligence on Allergy Diagnosis and Treatment.

Khan M, Banerjee S, Muskawad S, Maity R, Chowdhury S, Ejaz R Curr Allergy Asthma Rep. 2024; 24(7):361-372.

PMID: 38954325 DOI: 10.1007/s11882-024-01152-y.

Clinical Applications of Machine Learning.

Mateussi N, Rogers M, Grimsley E, Read M, Parikh R, Pietrobon R Ann Surg Open. 2024; 5(2):e423.

PMID: 38911656 PMC: 11191915. DOI: 10.1097/AS9.0000000000000423.

Developing an Inpatient Electronic Medical Record Phenotype for Hospital-Acquired Pressure Injuries: Case Study Using Natural Language Processing Models.

Nurmambetova E, Pan J, Zhang Z, Wu G, Lee S, Southern D JMIR AI. 2024; 2:e41264.

PMID: 38875552 PMC: 11041460. DOI: 10.2196/41264.

References

Murff H, Forster A, Peterson J, Fiskio J, Heiman H, Bates D . Electronically screening discharge summaries for adverse medical events. J Am Med Inform Assoc. 2003; 10(4):339-50. PMC: 181984. DOI: 10.1197/jamia.M1201. View

Bates D, Cullen D, Laird N, Petersen L, Small S, Servi D . Incidence of adverse drug events and potential adverse drug events. Implications for prevention. ADE Prevention Study Group. JAMA. 1995; 274(1):29-34. View

Bates D, Evans R, Murff H, Stetson P, Pizziferri L, Hripcsak G . Detecting adverse events using information technology. J Am Med Inform Assoc. 2003; 10(2):115-28. PMC: 150365. DOI: 10.1197/jamia.m1074. View

SAGER N, Lyman M, Nhan N, TICK L . Medical language processing: applications to patient data representation and automatic encoding. Methods Inf Med. 1995; 34(1-2):140-6. View

Brennan T, Localio A, Leape L, Laird N, Peterson L, Hiatt H . Identification of adverse events occurring during hospitalization. A cross-sectional study of litigation, quality assurance, and medical records at two teaching hospitals. Ann Intern Med. 1990; 112(3):221-6. DOI: 10.7326/0003-4819-112-3-221. View

Honigman B, Lee J, Rothschild J, Light P, Pulling R, Yu T . Using computerized data to identify adverse drug events in outpatients. J Am Med Inform Assoc. 2001; 8(3):254-66. PMC: 131033. DOI: 10.1136/jamia.2001.0080254. View

Mendonca E, Haas J, Shagina L, Larson E, Friedman C . Extracting information on pneumonia in infants using natural language processing of radiology reports. J Biomed Inform. 2005; 38(4):314-21. DOI: 10.1016/j.jbi.2005.02.003. View

Hripcsak G, Kuperman G, Friedman C . Extracting findings from narrative reports: software transferability and sources of physician disagreement. Methods Inf Med. 1998; 37(1):1-7. View

von Laue N, Schwappach D, Koeck C . The epidemiology of medical errors: a review of the literature. Wien Klin Wochenschr. 2003; 115(10):318-25. DOI: 10.1007/BF03041483. View

10.

Benson M, Junger A, Michel A, Sciuk G, Quinzio L, Marquardt K . Comparison of manual and automated documentation of adverse events with an Anesthesia Information Management System (AIMS). Stud Health Technol Inform. 2001; 77:925-9. View

11.

Field T, Gurwitz J, Harrold L, Rothschild J, DeBellis K, Seger A . Strategies for detecting adverse drug events among older persons in the ambulatory setting. J Am Med Inform Assoc. 2004; 11(6):492-8. PMC: 524629. DOI: 10.1197/jamia.M1586. View

12.

Fiszman M, Chapman W, Aronsky D, Evans R, Haug P . Automatic detection of acute bacterial pneumonia from chest X-ray reports. J Am Med Inform Assoc. 2000; 7(6):593-604. PMC: 129668. DOI: 10.1136/jamia.2000.0070593. View

13.

Forster A, Andrade J, van Walraven C . Validation of a discharge summary term search method to detect adverse events. J Am Med Inform Assoc. 2004; 12(2):200-6. PMC: 551551. DOI: 10.1197/jamia.M1653. View

14.

Benson M, Junger A, Fuchs C, Quinzio L, Bottger S, Jost A . Using an anesthesia information management system to prove a deficit in voluntary reporting of adverse events in a quality assurance program. J Clin Monit Comput. 2003; 16(3):211-7. DOI: 10.1023/a:1009977917319. View

15.

Hripcsak G, Friedman C, Alderson P, DuMouchel W, Johnson S, Clayton P . Unlocking clinical data from narrative reports: a study of natural language processing. Ann Intern Med. 1995; 122(9):681-8. DOI: 10.7326/0003-4819-122-9-199505010-00007. View

16.

Gundersen M, Haug P, Pryor T, Van Bree R, Koehler S, Bauer K . Development and evaluation of a computerized admission diagnoses encoding system. Comput Biomed Res. 1996; 29(5):351-72. DOI: 10.1006/cbmr.1996.0026. View

17.

Cao H, Stetson P, Hripcsak G . Assessing explicit error reporting in the narrative electronic medical record using keyword searching. J Biomed Inform. 2003; 36(1-2):99-105. DOI: 10.1016/s1532-0464(03)00058-3. View

18.

Cullen D, Sweitzer B, Bates D, Burdick E, Edmondson A, Leape L . Preventable adverse drug events in hospitalized patients: a comparative study of intensive care and general care units. Crit Care Med. 1997; 25(8):1289-97. DOI: 10.1097/00003246-199708000-00014. View

19.

Murff H, Patel V, Hripcsak G, Bates D . Detecting adverse events for patient safety research: a review of current methodologies. J Biomed Inform. 2003; 36(1-2):131-43. DOI: 10.1016/j.jbi.2003.08.003. View

20.

Jha A, Kuperman G, Rittenberg E, Teich J, Bates D . Identifying hospital admissions due to adverse drug events using a computer-based monitor. Pharmacoepidemiol Drug Saf. 2001; 10(2):113-9. DOI: 10.1002/pds.568. View