Inter-rater Agreement in Physician-coded Problem Lists

Overview

Journal AMIA Annu Symp Proc

Specialty Medical Informatics

Date 2006 Jun 17

PMID 16779119

Citations 12

Authors

Adam S Rothschild

Harold P Lehmann

George Hripcsak

Affiliations

Soon will be listed here.

Abstract

Coded problem lists will be increasingly used for many purposes in healthcare. The usefulness of coded problem lists may be limited by 1) how consistently clinicians enumerate patients' problems and 2) how consistently clinicians choose a given concept from a controlled terminology to represent a given problem. In this study, 10 physicians reviewed the same 5 clinical cases and created a coded problem list for each case using UMLS as a controlled terminology. We assessed inter-rater agreement for coded problem lists by computing the average pair-wise positive specific agreement for each case for all 10 reviewers. We also standardized problems to common terms across reviewers' lists for a given case, adjusting sequentially for synonymy, granularity, and general concept representation. Our results suggest that inter-rater agreement in unstandardized problem lists is moderate at best; standardization improves agreement, but much variability may be attributable to differences in clinicians' style and the inherent fuzziness of medical diagnosis.

Citing Articles

Data gaps in electronic health record (EHR) systems: An audit of problem list completeness during the COVID-19 pandemic.

Poulos J, Zhu L, Shah A Int J Med Inform. 2021; 150:104452.

PMID: 33864979 PMC: 9759969. DOI: 10.1016/j.ijmedinf.2021.104452.

Performance of a computable phenotype for pediatric asthma using the problem list.

Tang M, Goldstein B, He J, Hurst J, Lang J Ann Allergy Asthma Immunol. 2020; 125(5):611-613.e1.

PMID: 32687988 PMC: 7606382. DOI: 10.1016/j.anai.2020.07.009.

Automating the Transformation of Free-Text Clinical Problems into SNOMED CT Expressions.

Peterson K, Liu H AMIA Jt Summits Transl Sci Proc. 2020; 2020:497-506.

PMID: 32477671 PMC: 7233039.

Recording problems and diagnoses in clinical care: developing guidance for healthcare professionals and system designers.

Shah A, Quinn N, Chaudhry A, Sullivan R, Costello J, ORiordan D BMJ Health Care Inform. 2019; 26(1).

PMID: 31874855 PMC: 7062352. DOI: 10.1136/bmjhci-2019-100106.

Characterizing the Utilization of the Problem List for Pediatric Trauma Care.

Durojaiye A, McGerorge N, Kristen W, Oruc C, Fackler J, Gurses A AMIA Annu Symp Proc. 2019; 2018:404-412.

PMID: 30815080 PMC: 6371241.

References

Elkin P, Tuttle M, Keck K, Campbell K, Atkin G, Chute C . The role of compositionality in standardized problem list generation. Stud Health Technol Inform. 1999; 52 Pt 1:660-4. View

Cimino J, Patel V, Kushniruk A . Studying the human-computer-terminology interface. J Am Med Inform Assoc. 2001; 8(2):163-73. PMC: 134555. DOI: 10.1136/jamia.2001.0080163. View

Scherpbier H, Abrams R, Roth D, Hail J . A simple approach to physician entry of patient problem list. Proc Annu Symp Comput Appl Med Care. 1994; :206-10. PMC: 2247970. View

Johnson K, George E . The rubber meets the road: integrating the Unified Medical Language System Knowledge Source Server into the computer-based patient record. Proc AMIA Annu Fall Symp. 1997; :17-21. PMC: 2233548. View

Weed L . Medical records that guide and teach. N Engl J Med. 1968; 278(11):593-600. DOI: 10.1056/NEJM196803142781105. View

Campbell J, Payne T . A comparison of four schemes for codification of problem lists. Proc Annu Symp Comput Appl Med Care. 1994; :201-5. PMC: 2247911. View

Szeto H, Coleman R, Gholami P, Hoffman B, Goldstein M . Accuracy of computerized outpatient diagnoses in a Veterans Affairs general medicine clinic. Am J Manag Care. 2002; 8(1):37-43. View

Warren J, Collins J, Sorrentino C, Campbell J . Just-in-time coding of the problem list in a clinical environment. Proc AMIA Symp. 1999; :280-4. PMC: 2232253. View

Wasserman H, Wang J . An applied evaluation of SNOMED CT as a clinical vocabulary for the computerized diagnosis and problem list. AMIA Annu Symp Proc. 2004; :699-703. PMC: 1479961. View

10.

Elkin P, Mohr D, Tuttle M, Cole W, Atkin G, Keck K . Standardized problem list generation, utilizing the Mayo canonical vocabulary embedded within the Unified Medical Language System. Proc AMIA Annu Fall Symp. 1997; :500-4. PMC: 2233586. View

11.

Rothschild A, Lehmann H . Information retrieval performance of probabilistically generated, problem-specific computerized provider order entry pick-lists: a pilot study. J Am Med Inform Assoc. 2005; 12(3):322-30. PMC: 1090464. DOI: 10.1197/jamia.M1670. View

12.

Cimino J . Desiderata for controlled medical vocabularies in the twenty-first century. Methods Inf Med. 1998; 37(4-5):394-403. PMC: 3415631. View

13.

Chute C, Elkin P, Fenton S, Atkin G . A clinical terminology in the post modern era: pragmatic problem list development. Proc AMIA Symp. 1999; :795-9. PMC: 2232227. View

14.

Wang S, Bates D, Chueh H, Karson A, Maviglia S, Greim J . Automated coded ambulatory problem lists: evaluation of a vocabulary and a data entry tool. Int J Med Inform. 2003; 72(1-3):17-28. DOI: 10.1016/j.ijmedinf.2003.08.002. View

15.

Brown S, Miller R, Camp H, Guise D, Walker H . Empirical derivation of an electronic clinically useful problem statement system. Ann Intern Med. 1999; 131(2):117-26. DOI: 10.7326/0003-4819-131-2-199907200-00008. View

16.

Goldberg H, Goldsmith D, Law V, Keck K, Tuttle M, Safran C . An evaluation of UMLS as a controlled terminology for the Problem List Toolkit. Stud Health Technol Inform. 1999; 52 Pt 1:609-12. View

17.

Miller J, Driscoll C, Kilpatrick S, Quillen Jr E . Management of prenatal care information: integration of the problem list and clinical comments. Top Health Inf Manage. 2003; 24(1):42-9. View

18.

Hripcsak G, Rothschild A . Agreement, the f-measure, and reliability in information retrieval. J Am Med Inform Assoc. 2005; 12(3):296-8. PMC: 1090460. DOI: 10.1197/jamia.M1733. View

19.

Safran C . Searching for answers on a clinical information system. Methods Inf Med. 1995; 34(1-2):79-84. View

20.

Hales J, Schoeffler K, KESSLER D . Extracting medical knowledge for a coded problem list vocabulary from the UMLS Knowledge Sources. Proc AMIA Symp. 1999; :275-9. PMC: 2232222. View