Are Three Methods Better Than One? A Comparative Assessment of Usability Evaluation Methods in an EHR

Overview

Journal Int J Med Inform

Specialty Medical Informatics

Date 2014 Mar 4

PMID 24583045

Citations 22

Authors

Muhammad F Walji

Elsbeth Kalenderian

Mark Piotrowski

Duong Tran

Krishna K Kookal

Oluwabunmi Tokede

Joel M White

Ram Vaderhobli

Rachel Ramoni

Paul C Stark

Nicole S Kimmes

Maxim Lagerweij

Vimla L Patel

Affiliations

Soon will be listed here.

Abstract

Objective: To comparatively evaluate the effectiveness of three different methods involving end-users for detecting usability problems in an EHR: user testing, semi-structured interviews and surveys.

Materials And Methods: Data were collected at two major urban dental schools from faculty, residents and dental students to assess the usability of a dental EHR for developing a treatment plan. These included user testing (N=32), semi-structured interviews (N=36), and surveys (N=35).

Results: The three methods together identified a total of 187 usability violations: 54% via user testing, 28% via the semi-structured interview and 18% from the survey method, with modest overlap. These usability problems were classified into 24 problem themes in 3 broad categories. User testing covered the broadest range of themes (83%), followed by the interview (63%) and survey (29%) methods.

Discussion: Multiple evaluation methods provide a comprehensive approach to identifying EHR usability challenges and specific problems. The three methods were found to be complementary, and thus each can provide unique insights for software enhancement. Interview and survey methods were found not to be sufficient by themselves, but when used in conjunction with the user testing method, they provided a comprehensive evaluation of the EHR.

Conclusion: We recommend using a multi-method approach when testing the usability of health information technology because it provides a more comprehensive picture of usability challenges.

Citing Articles

Usability of a cross-system housing stability program for juvenile courts: A multimethod study of probation counselor perspectives.

Bishop A, Walker S, Baumgarten E, Vick K, Dean T Implement Res Pract. 2025; 6:26334895251319814.

PMID: 39991469 PMC: 11843703. DOI: 10.1177/26334895251319814.

Evaluating and improving the usability of a mHealth platform to assess postoperative dental pain.

Ibarra-Noriega A, Yansane A, Mullins J, Simmons K, Skourtes N, Holmes D JAMIA Open. 2024; 7(1):ooae018.

PMID: 38476372 PMC: 10928307. DOI: 10.1093/jamiaopen/ooae018.

Towards co-design of rehabilitation technologies: a collaborative approach to prioritize usability issues.

Clanchy K, Mitchell J, Mulholland K, Jurd E, Kendall E, Lloyd D Front Rehabil Sci. 2024; 5:1302179.

PMID: 38450206 PMC: 10915061. DOI: 10.3389/fresc.2024.1302179.

An Online Psychological Program for Adolescents and Young Adults With Headaches: Iterative Design and Rapid Usability Testing.

Huguet A, Rozario S, Wozney L, McGrath P JMIR Hum Factors. 2023; 10:e48677.

PMID: 38085567 PMC: 10751633. DOI: 10.2196/48677.

Current state of dental informatics in the field of health information systems: a scoping review.

Benoit B, Frederic B, Jean-Charles D BMC Oral Health. 2022; 22(1):131.

PMID: 35439988 PMC: 9020044. DOI: 10.1186/s12903-022-02163-9.

References

Kemper A, Uren R, Clark S . Adoption of electronic health records in primary care pediatric practices. Pediatrics. 2006; 118(1):e20-4. DOI: 10.1542/peds.2005-3000. View

Menachemi N, Langley A, Brooks R . The use of information technologies among rural and urban physicians in Florida. J Med Syst. 2007; 31(6):483-8. DOI: 10.1007/s10916-007-9088-6. View

Khajouei R, Peute L, Hasman A, Jaspers M . Classification and prioritization of usability problems using an augmented classification scheme. J Biomed Inform. 2011; 44(6):948-57. DOI: 10.1016/j.jbi.2011.07.002. View

Kushniruk A, Triola M, Borycki E, Stein B, Kannry J . Technology induced error and usability: the relationship between usability problems and prescription errors when using a handheld application. Int J Med Inform. 2005; 74(7-8):519-26. DOI: 10.1016/j.ijmedinf.2005.01.003. View

Horsky J, Kuperman G, Patel V . Comprehensive analysis of a medication dosing error related to CPOE. J Am Med Inform Assoc. 2005; 12(4):377-82. PMC: 1174881. DOI: 10.1197/jamia.M1740. View

Koppel R, Metlay J, Cohen A, Abaluck B, Russell Localio A, Kimmel S . Role of computerized physician order entry systems in facilitating medication errors. JAMA. 2005; 293(10):1197-203. DOI: 10.1001/jama.293.10.1197. View

Bates D, Cohen M, Leape L, Overhage J, Shabot M, Sheridan T . Reducing the frequency of errors in medicine using information technology. J Am Med Inform Assoc. 2001; 8(4):299-308. PMC: 130074. DOI: 10.1136/jamia.2001.0080299. View

Chan J, Shojania K, Easty A, Etchells E . Usability evaluation of order sets in a computerised provider order entry system. BMJ Qual Saf. 2011; 20(11):932-40. DOI: 10.1136/bmjqs.2010.050021. View

Hill H, Stewart D, Ash J . Health Information Technology Systems profoundly impact users: a case study in a dental school. J Dent Educ. 2010; 74(4):434-45. View

10.

Nykanen P, Brender J, Talmon J, de Keizer N, Rigby M, Beuscart-Zephir M . Guideline for good evaluation practice in health informatics (GEP-HI). Int J Med Inform. 2011; 80(12):815-27. DOI: 10.1016/j.ijmedinf.2011.08.004. View

11.

Hillestad R, Bigelow J, Bower A, Girosi F, Meili R, Scoville R . Can electronic medical record systems transform health care? Potential health benefits, savings, and costs. Health Aff (Millwood). 2005; 24(5):1103-17. DOI: 10.1377/hlthaff.24.5.1103. View

12.

Zhang J . Human-centered computing in health information systems part 2: evaluation. J Biomed Inform. 2005; 38(3):173-5. DOI: 10.1016/j.jbi.2004.12.005. View

13.

Kalenderian E, Ramoni R, White J, Schoonheim-Klein M, Stark P, Kimmes N . The development of a dental diagnostic terminology. J Dent Educ. 2011; 75(1):68-76. PMC: 3107733. View

14.

Walji M, Kalenderian E, Tran D, Kookal K, Nguyen V, Tokede O . Detection and characterization of usability problems in structured data entry interfaces in dentistry. Int J Med Inform. 2012; 82(2):128-38. PMC: 3492538. DOI: 10.1016/j.ijmedinf.2012.05.018. View

15.

Middleton B, Bloomrosen M, Dente M, Hashmat B, Koppel R, Overhage J . Enhancing patient safety and quality of care by improving the usability of electronic health record systems: recommendations from AMIA. J Am Med Inform Assoc. 2013; 20(e1):e2-8. PMC: 3715367. DOI: 10.1136/amiajnl-2012-001458. View

16.

Johnson C, Nahm M, Shaw R, Dunham A, Newby K, Dolor R . Can prospective usability evaluation predict data errors?. AMIA Annu Symp Proc. 2011; 2010:346-50. PMC: 3041437. View

17.

Horsky J, Zhang J, Patel V . To err is not entirely human: complex technology and user cognition. J Biomed Inform. 2005; 38(4):264-6. DOI: 10.1016/j.jbi.2005.05.002. View

18.

Valdes I, Kibbe D, Tolleson G, Kunik M, Petersen L . Barriers to proliferation of electronic medical records. Inform Prim Care. 2004; 12(1):3-9. DOI: 10.14236/jhi.v12i1.102. View

19.

Viitanen J, Hypponen H, Laaveri T, Vanska J, Reponen J, Winblad I . National questionnaire study on clinical ICT systems proofs: physicians suffer from poor usability. Int J Med Inform. 2011; 80(10):708-25. DOI: 10.1016/j.ijmedinf.2011.06.010. View

20.

Campbell E, Sittig D, Ash J, Guappone K, Dykstra R . Types of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc. 2006; 13(5):547-56. PMC: 1561794. DOI: 10.1197/jamia.M2042. View