Evaluation of a Clinical Decision Support System for Rare Diseases: a Qualitative Study

Overview

Journal BMC Med Inform Decis Mak

Publisher Biomed Central

Specialty Medical Informatics

Date 2021 Feb 19

PMID 33602191

Citations 13

Authors

Jannik Schaaf

Martin Sedlmayr

Brita Sedlmayr

Hans-Ulrich Prokosch

Holger Storf

Affiliations

Soon will be listed here.

Abstract

Background: Rare Diseases (RDs) are difficult to diagnose. Clinical Decision Support Systems (CDSS) could support the diagnosis for RDs. The Medical Informatics in Research and Medicine (MIRACUM) consortium developed a CDSS for RDs based on distributed clinical data from eight German university hospitals. To support the diagnosis for difficult patient cases, the CDSS uses data from the different hospitals to perform a patient similarity analysis to obtain an indication of a diagnosis. To optimize our CDSS, we conducted a qualitative study to investigate usability and functionality of our designed CDSS.

Methods: We performed a Thinking Aloud Test (TA-Test) with RDs experts working in Rare Diseases Centers (RDCs) at MIRACUM locations which are specialized in diagnosis and treatment of RDs. An instruction sheet with tasks was prepared that the participants should perform with the CDSS during the study. The TA-Test was recorded on audio and video, whereas the resulting transcripts were analysed with a qualitative content analysis, as a ruled-guided fixed procedure to analyse text-based data. Furthermore, a questionnaire was handed out at the end of the study including the System Usability Scale (SUS).

Results: A total of eight experts from eight MIRACUM locations with an established RDC were included in the study. Results indicate that more detailed information about patients, such as descriptive attributes or findings, can help the system perform better. The system was rated positively in terms of functionality, such as functions that enable the user to obtain an overview of similar patients or medical history of a patient. However, there is a lack of transparency in the results of the CDSS patient similarity analysis. The study participants often stated that the system should present the user with an overview of exact symptoms, diagnosis, and other characteristics that define two patients as similar. In the usability section, the CDSS received a score of 73.21 points, which is ranked as good usability.

Conclusions: This qualitative study investigated the usability and functionality of a CDSS of RDs. Despite positive feedback about functionality of system, the CDSS still requires some revisions and improvement in transparency of the patient similarity analysis.

Citing Articles

Evaluating and Enhancing the Fitness-for-Purpose of Electronic Health Record Data: Qualitative Study on Current Practices and Pathway to an Automated Approach Within the Medical Informatics for Research and Care in University Medicine Consortium.

Kamdje Wabo G, Moorthy P, Siegel F, Seuchter S, Ganslandt T JMIR Med Inform. 2024; 12:e57153.

PMID: 39158950 PMC: 11369535. DOI: 10.2196/57153.

Gynaecological Artificial Intelligence Diagnostics (GAID) GAID and Its Performance as a Tool for the Specialist Doctor.

Tanos P, Yiangou I, Prokopiou G, Kakas A, Tanos V Healthcare (Basel). 2024; 12(2).

PMID: 38255110 PMC: 10815463. DOI: 10.3390/healthcare12020223.

Web-based eHealth Clinical Decision Support System as a tool for the treat-to-target management of patients with systemic lupus erythematosus: .

Parra Sanchez A, Grimberg M, Hanssen M, Aben M, Jairth E, Dhoeme P BMJ Health Care Inform. 2023; 30(1).

PMID: 37751942 PMC: 10533702. DOI: 10.1136/bmjhci-2023-100811.

Requirements analysis for an AI-based clinical decision support system for general practitioners: a user-centered design process.

Schutze D, Holtz S, Neff M, Kohler S, Schaaf J, Frischen L BMC Med Inform Decis Mak. 2023; 23(1):144.

PMID: 37525175 PMC: 10391889. DOI: 10.1186/s12911-023-02245-w.

Ten Topics to Get Started in Medical Informatics Research.

Wolfien M, Ahmadi N, Fitzer K, Grummt S, Heine K, Jung I J Med Internet Res. 2023; 25:e45948.

PMID: 37486754 PMC: 10407648. DOI: 10.2196/45948.

References

Lopes M, Koch V, Sarrubbi-Junior V, Gallo P, Carneiro-Sampaio M . Difficulties in the diagnosis and treatment of rare diseases according to the perceptions of patients, relatives and health care professionals. Clinics (Sao Paulo). 2018; 73:e68. PMC: 5866403. DOI: 10.6061/clinics/2018/e68. View

Richardson S, Mishuris R, OConnell A, Feldstein D, Hess R, Smith P . "Think aloud" and "Near live" usability testing of two complex clinical decision support tools. Int J Med Inform. 2017; 106:1-8. PMC: 5679128. DOI: 10.1016/j.ijmedinf.2017.06.003. View

Bezemer T, De Groot M, Blasse E, Ten Berg M, Kappen T, Bredenoord A . A Human(e) Factor in Clinical Decision Support Systems. J Med Internet Res. 2019; 21(3):e11732. PMC: 6444220. DOI: 10.2196/11732. View

Li A, Kannry J, Kushniruk A, Chrimes D, McGinn T, Edonyabo D . Integrating usability testing and think-aloud protocol analysis with "near-live" clinical simulations in evaluating clinical decision support. Int J Med Inform. 2012; 81(11):761-72. DOI: 10.1016/j.ijmedinf.2012.02.009. View

Chrimes D, Kitos N, Kushniruk A, Mann D . Usability testing of Avoiding Diabetes Thru Action Plan Targeting (ADAPT) decision support for integrating care-based counseling of pre-diabetes in an electronic health record. Int J Med Inform. 2014; 83(9):636-47. PMC: 4212327. DOI: 10.1016/j.ijmedinf.2014.05.002. View

Press A, McCullagh L, Khan S, Schachter A, Pardo S, McGinn T . Usability Testing of a Complex Clinical Decision Support Tool in the Emergency Department: Lessons Learned. JMIR Hum Factors. 2016; 2(2):e14. PMC: 4797671. DOI: 10.2196/humanfactors.4537. View

Taruscio D, Floridia G, Salvatore M, Groft S, Gahl W . Undiagnosed Diseases: Italy-US Collaboration and International Efforts to Tackle Rare and Common Diseases Lacking a Diagnosis. Adv Exp Med Biol. 2017; 1031:25-38. DOI: 10.1007/978-3-319-67144-4_2. View

Schaaf J, Prokosch H, Boeker M, Schaefer J, Vasseur J, Storf H . Interviews with experts in rare diseases for the development of clinical decision support system software - a qualitative study. BMC Med Inform Decis Mak. 2020; 20(1):230. PMC: 7493382. DOI: 10.1186/s12911-020-01254-3. View

Berner E . Diagnostic decision support systems: why aren't they used more and what can we do about it?. AMIA Annu Symp Proc. 2007; :1167-8. PMC: 1839633. View

10.

Rahimi B, Nadri H, Lotfnezhad Afshar H, Timpka T . A Systematic Review of the Technology Acceptance Model in Health Informatics. Appl Clin Inform. 2018; 9(3):604-634. PMC: 6094026. DOI: 10.1055/s-0038-1668091. View

11.

OBrien B, Harris I, Beckman T, Reed D, Cook D . Standards for reporting qualitative research: a synthesis of recommendations. Acad Med. 2014; 89(9):1245-51. DOI: 10.1097/ACM.0000000000000388. View

12.

Schaaf J, Sedlmayr M, Prokosch H, Ganslandt T, Schade-Brittinger C, von Wagner M . The Status Quo of Rare Diseases Centres for the Development of a Clinical Decision Support System - A Cross-Sectional Study. Stud Health Technol Inform. 2020; 271:176-183. DOI: 10.3233/SHTI200094. View

13.

Joe J, Chaudhuri S, Le T, Thompson H, Demiris G . The use of think-aloud and instant data analysis in evaluation research: Exemplar and lessons learned. J Biomed Inform. 2015; 56:284-91. PMC: 4532606. DOI: 10.1016/j.jbi.2015.06.001. View

14.

Griffon N, Schuers M, Dhombres F, Merabti T, Kerdelhue G, Rollin L . Searching for rare diseases in PubMed: a blind comparison of Orphanet expert query and query based on terminological knowledge. BMC Med Inform Decis Mak. 2016; 16:101. PMC: 4970261. DOI: 10.1186/s12911-016-0333-0. View

15.

Storf H, Schaaf J, Kadioglu D, Gobel J, Wagner T, Uckert F . [Registries for rare diseases : OSSE - An open-source framework for technical implementation]. Bundesgesundheitsblatt Gesundheitsforschung Gesundheitsschutz. 2017; 60(5):523-531. DOI: 10.1007/s00103-017-2536-7. View

16.

Sutton R, Pincock D, Baumgart D, Sadowski D, Fedorak R, Kroeker K . An overview of clinical decision support systems: benefits, risks, and strategies for success. NPJ Digit Med. 2020; 3:17. PMC: 7005290. DOI: 10.1038/s41746-020-0221-y. View

17.

Blecker S, Pandya R, Stork S, Mann D, Kuperman G, Shelley D . Interruptive Versus Noninterruptive Clinical Decision Support: Usability Study. JMIR Hum Factors. 2019; 6(2):e12469. PMC: 6492060. DOI: 10.2196/12469. View

18.

Shen F, Liu S, Wang Y, Wen A, Wang L, Liu H . Utilization of Electronic Medical Records and Biomedical Literature to Support the Diagnosis of Rare Diseases Using Data Fusion and Collaborative Filtering Approaches. JMIR Med Inform. 2018; 6(4):e11301. PMC: 6231873. DOI: 10.2196/11301. View

19.

Miller A, Moon B, Anders S, Walden R, Brown S, Montella D . Integrating computerized clinical decision support systems into clinical work: A meta-synthesis of qualitative research. Int J Med Inform. 2015; 84(12):1009-18. DOI: 10.1016/j.ijmedinf.2015.09.005. View

20.

LeRouge C, Wickramasinghe N . A review of user-centered design for diabetes-related consumer health informatics technologies. J Diabetes Sci Technol. 2013; 7(4):1039-56. PMC: 3879771. DOI: 10.1177/193229681300700429. View