Measuring the Impact of Diagnostic Decision Support on the Quality of Clinical Decision Making: Development of a Reliable and Valid Composite Score

Overview

Journal J Am Med Inform Assoc

Publisher Oxford University Press

Specialty Medical Informatics

Date 2003 Aug 20

PMID 12925549

Citations 19

Authors

Padmanabhan Ramnarayan

Ritika R Kapoor

Michael Coren

Vasantha Nanduri

Amanda L Tomlinson

Paul M Taylor

Jeremy C Wyatt

Joseph F Britto

Affiliations

Soon will be listed here.

Abstract

Objective: Few previous studies evaluating the benefits of diagnostic decision support systems have simultaneously measured changes in diagnostic quality and clinical management prompted by use of the system. This report describes a reliable and valid scoring technique to measure the quality of clinical decision plans in an acute medical setting, where diagnostic decision support tools might prove most useful.

Design: Sets of differential diagnoses and clinical management plans generated by 71 clinicians for six simulated cases, before and after decision support from a Web-based pediatric differential diagnostic tool (ISABEL), were used.

Measurements: A composite quality score was calculated separately for each diagnostic and management plan by considering the appropriateness value of each component diagnostic or management suggestion, a weighted sum of individual suggestion ratings, relevance of the entire plan, and its comprehensiveness. The reliability and validity (face, concurrent, construct, and content) of these two final scores were examined.

Results: Two hundred fifty-two diagnostic and 350 management suggestions were included in the interrater reliability analysis. There was good agreement between raters (intraclass correlation coefficient, 0.79 for diagnoses, and 0.72 for management). No counterintuitive scores were demonstrated on visual inspection of the sets. Content validity was verified by a consultation process with pediatricians. Both scores discriminated adequately between the plans of consultants and medical students and correlated well with clinicians' subjective opinions of overall plan quality (Spearman rho 0.65, p < 0.01). The diagnostic and management scores for each episode showed moderate correlation (r = 0.51).

Conclusion: The scores described can be used as key outcome measures in a larger study to fully assess the value of diagnostic decision aids, such as the ISABEL system.

Citing Articles

Can clinical decision support systems be an asset in medical education? An experimental approach.

Kafke S, Kuhlmey A, Schuster J, Bluher S, Czimmeck C, Zoellick J BMC Med Educ. 2023; 23(1):570.

PMID: 37568144 PMC: 10416486. DOI: 10.1186/s12909-023-04568-8.

Assessing the Performance of a New Artificial Intelligence-Driven Diagnostic Support Tool Using Medical Board Exam Simulations: Clinical Vignette Study.

Ben-Shabat N, Sloma A, Weizman T, Kiderman D, Amital H JMIR Med Inform. 2021; 9(11):e32507.

PMID: 34672262 PMC: 8672291. DOI: 10.2196/32507.

Improving Team-Based Decision Making Using Data Analytics and Informatics: Protocol for a Collaborative Decision Support Design.

Roosan D, Law A, Karim M, Roosan M JMIR Res Protoc. 2019; 8(11):e16047.

PMID: 31774412 PMC: 6906625. DOI: 10.2196/16047.

A review of measurement practice in studies of clinical decision support systems 1998-2017.

Scott P, Brown A, Adedeji T, Wyatt J, Georgiou A, Eisenstein E J Am Med Inform Assoc. 2019; 26(10):1120-1128.

PMID: 30990522 PMC: 6748820. DOI: 10.1093/jamia/ocz035.

Adverse event detection by medical record review is reproducible, but the assessment of their preventability is not.

Klein D, Rennenberg R, Koopmans R, Prins M PLoS One. 2018; 13(11):e0208087.

PMID: 30496243 PMC: 6264838. DOI: 10.1371/journal.pone.0208087.

References

Wexler J, Swender P, Tunnessen Jr W, Oski F . Impact of a system of computer-assisted diagnosis. Initial evaluation of the hospitalized patient. Am J Dis Child. 1975; 129(2):203-5. DOI: 10.1001/archpedi.1975.02120390037008. View

de Dombal F, Leaper D, Staniland J, McCann A, Horrocks J . Computer-aided diagnosis of acute abdominal pain. Br Med J. 1972; 2(5804):9-13. PMC: 1789017. DOI: 10.1136/bmj.2.5804.9. View

Miller R, Pople Jr H, Myers J . Internist-1, an experimental computer-based diagnostic consultant for general internal medicine. N Engl J Med. 1982; 307(8):468-76. DOI: 10.1056/NEJM198208193070803. View

Pozen M, DAgostino R, Selker H, Sytkowski P, Hood Jr W . A predictive instrument to improve coronary-care-unit admission practices in acute ischemic heart disease. A prospective multicenter clinical trial. N Engl J Med. 1984; 310(20):1273-8. DOI: 10.1056/NEJM198405173102001. View

Kent D, Shortliffe E, Carlson R, Bischoff M, Jacobs C . Improvements in data collection through physician use of a computer-based chemotherapy treatment consultant. J Clin Oncol. 1985; 3(10):1409-17. DOI: 10.1200/JCO.1985.3.10.1409. View

Nelson S, Blois M, Tuttle M, Erlbaum M, Harrison P, Kim H . Evaluating RECONSIDER. A computer program for diagnostic prompting. J Med Syst. 1985; 9(5-6):379-88. DOI: 10.1007/BF00992575. View

Miller P . Extending computer-based critiquing to a new domain: ATTENDING, ESSENTIAL-ATTENDING, and VQ-ATTENDING. Int J Clin Monit Comput. 1986; 2(3):135-42. DOI: 10.1007/BF02915880. View

Adams I, Chan M, Clifford P, Cooke W, DALLOS V, de Dombal F . Computer aided diagnosis of acute abdominal pain: a multicentre study. Br Med J (Clin Res Ed). 1986; 293(6550):800-4. PMC: 1341582. DOI: 10.1136/bmj.293.6550.800. View

Miller R, Masarie F, Myers J . Quick medical reference (QMR) for diagnostic assistance. MD Comput. 1986; 3(5):34-48. View

10.

Barnett G, Cimino J, Hupp J, Hoffer E . DXplain. An evolving diagnostic decision-support system. JAMA. 1987; 258(1):67-74. DOI: 10.1001/jama.258.1.67. View

11.

Dawson N, Arkes H, Siciliano C, Blinkhorn R, Lakshmanan M, Petrelli M . Hindsight bias: an impediment to accurate probability estimation in clinicopathologic conferences. Med Decis Making. 1988; 8(4):259-64. DOI: 10.1177/0272989X8800800406. View

12.

Bankowitz R, McNeil M, Challinor S, Parker R, Kapoor W, Miller R . A computer-assisted medical diagnostic consultation service. Implementation and prospective evaluation of a prototype. Ann Intern Med. 1989; 110(10):824-32. DOI: 10.7326/0003-4819-110-10-824. View

13.

Bankowitz R, McNeil M, Challinor S, Miller R . Effect of a computer-assisted general medicine diagnostic consultation service on housestaff diagnostic strategy. Methods Inf Med. 1989; 28(4):352-6. View

14.

Miller R, Masarie Jr F . The demise of the "Greek Oracle" model for medical diagnostic systems. Methods Inf Med. 1990; 29(1):1-2. View

15.

van der Lei J, Musen M, van der Does E, Man in t Veld A, van Bemmel J . Comparison of computer-aided and human review of general practitioners' management of hypertension. Lancet. 1991; 338(8781):1504-8. DOI: 10.1016/0140-6736(91)92311-o. View

16.

Wellwood J, Johannessen S, Spiegelhalter D . How does computer-aided diagnosis improve the management of acute abdominal pain?. Ann R Coll Surg Engl. 1992; 74(1):40-6. PMC: 2497469. View

17.

Bankowitz R, Lave J, McNeil M . A method for assessing the impact of a computer-based decision support system on health care outcomes. Methods Inf Med. 1992; 31(1):3-10. View

18.

Berner E, Webster G, Shugerman A, Jackson J, Algina J, Baker A . Performance of four computer-based diagnostic systems. N Engl J Med. 1994; 330(25):1792-6. DOI: 10.1056/NEJM199406233302506. View

19.

Haug P, Gardner R, Tate K, Evans R, East T, Kuperman G . Decision support in medicine: examples from the HELP system. Comput Biomed Res. 1994; 27(5):396-418. DOI: 10.1006/cbmr.1994.1030. View

20.

Miller R . Medical diagnostic decision support systems--past, present, and future: a threaded bibliography and brief commentary. J Am Med Inform Assoc. 1994; 1(1):8-27. PMC: 116181. DOI: 10.1136/jamia.1994.95236141. View