The ABCs of DKA: Development and Validation of a Computer-Based Simulator and Scoring System

Overview

Journal J Gen Intern Med

Publisher Springer

Specialty General Medicine

Date 2015 Jul 16

PMID 26173518

Citations 3

Authors

Catherine H Y Yu

Sharon Straus

Ryan Brydges

Affiliations

Soon will be listed here.

Abstract

Background: Clinical management of diabetic ketoacidosis (DKA) continues to be suboptimal; simulation-based training may bridge this gap and is particularly applicable to teaching DKA management skills given it enables learning of basic knowledge, as well as clinical reasoning and patient management skills.

Objectives: 1) To develop, test, and refine a computer-based simulator of DKA management; 2) to collect validity evidence, according to National Standard's validity framework; and 3) to judge whether the simulator scoring system is an appropriate measure of DKA management skills of undergraduate and postgraduate medical trainees.

Design: After developing the DKA simulator, we completed usability testing to optimize its functionality. We then conducted a preliminary validation of the scoring system for measuring trainees' DKA management skills.

Participants: We recruited year 1 and year 3 medical students, year 2 postgraduate trainees, and endocrinologists (n = 75); each completed a simulator run, and we collected their simulator-computed scores.

Main Measures: We collected validity evidence related to content, internal structure, relations with other variables, and consequences.

Key Results: Our simulator consists of six cases highlighting DKA management priorities. Real-time progression of each case includes interactive order entry, laboratory and clinical data, and individualised feedback. Usability assessment identified issues with clarity of system status, user control, efficiency of use, and error prevention. Regarding validity evidence, Cronbach's α was 0.795 for the seven subscales indicating favorable internal structure evidence. Participants' scores showed a significant effect of training level (p < 0.001). Scores also correlated with the number of DKA patients they reported treating, weeks on Medicine rotation, and comfort with managing DKA. A score on the simulation exercise of 75 % had a sensitivity and specificity of 94.7 % and 51.8%, respectively, for delineating between expert staff physicians and trainees.

Conclusions: We demonstrate how a simulator and scoring system can be developed, tested, and refined to determine its quality for use as an assessment modality. Our evidence suggests that it can be used for formative assessment of trainees' DKA management skills.

Citing Articles

Evidence of Construct Validity of Computer-Based Tests for Clinical Reasoning: Instrument Validation Study.

Zuo T, Sun B, Guan X, Zheng B, Qu B JMIR Serious Games. 2021; 9(4):e17670.

PMID: 34751658 PMC: 8663660. DOI: 10.2196/17670.

Clinical Conundrums in the Management of Diabetic Ketoacidosis in the Elderly.

Sehgal V, Ulmer B J Transl Int Med. 2019; 7(1):10-14.

PMID: 30997351 PMC: 6463825. DOI: 10.2478/jtim-2019-0003.

A systematic review of serious games in medical education: quality of evidence and pedagogical strategy.

Gorbanev I, Agudelo-Londono S, Gonzalez R, Cortes A, Pomares A, Delgadillo V Med Educ Online. 2018; 23(1):1438718.

PMID: 29457760 PMC: 5827764. DOI: 10.1080/10872981.2018.1438718.

References

Ericsson K . Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad Med. 2004; 79(10 Suppl):S70-81. DOI: 10.1097/00001888-200410001-00022. View

Issenberg S, McGaghie W, Petrusa E, Gordon D, Scalese R . Features and uses of high-fidelity medical simulations that lead to effective learning: a BEME systematic review. Med Teach. 2005; 27(1):10-28. DOI: 10.1080/01421590500046924. View

Cook D, Beckman T . Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med. 2006; 119(2):166.e7-16. DOI: 10.1016/j.amjmed.2005.10.036. View

Triola M, Feldman H, Kalet A, Zabar S, Kachur E, Gillespie C . A randomized trial of teaching clinical skills using virtual and live standardized patients. J Gen Intern Med. 2006; 21(5):424-9. PMC: 1484797. DOI: 10.1111/j.1525-1497.2006.00421.x. View

Volkova N, Fletcher C, Tevendale R, Munyaradzi S, Elliot S, Peterson M . Impact of a multidisciplinary approach to guideline implementation in diabetic ketoacidosis. Am J Med Qual. 2008; 23(1):47-55. DOI: 10.1177/1062860607311015. View

Ferreri R . Treatment practices of diabetic ketoacidosis at a large teaching hospital. J Nurs Care Qual. 2008; 23(2):147-54. DOI: 10.1097/01.NCQ.0000313764.64149.6d. View

Michelson J, Manning L . Competency assessment in simulation-based procedural education. Am J Surg. 2008; 196(4):609-15. DOI: 10.1016/j.amjsurg.2007.09.050. View

Pusic M, Pecaric M, Boutis K . How much practice is enough? Using learning curves to assess the deliberate practice of radiograph interpretation. Acad Med. 2011; 86(6):731-6. DOI: 10.1097/ACM.0b013e3182178c3c. View

Pusic M, Kessler D, Szyld D, Kalet A, Pecaric M, Boutis K . Experience curves as an organizing framework for deliberate practice in emergency medicine learning. Acad Emerg Med. 2012; 19(12):1476-80. DOI: 10.1111/acem.12043. View

10.

Cook D, Brydges R, Zendejas B, Hamstra S, Hatala R . Technology-enhanced simulation to assess health professionals: a systematic review of validity evidence, research methods, and reporting quality. Acad Med. 2013; 88(6):872-83. DOI: 10.1097/ACM.0b013e31828ffdcf. View

11.

Cheng A . Canadian Diabetes Association 2013 clinical practice guidelines for the prevention and management of diabetes in Canada. Introduction. Can J Diabetes. 2013; 37 Suppl 1:S1-3. DOI: 10.1016/j.jcjd.2013.01.009. View

12.

McGaghie W, Issenberg S, Barsuk J, Wayne D . A critical review of simulation-based mastery learning with translational outcomes. Med Educ. 2014; 48(4):375-85. DOI: 10.1111/medu.12391. View

13.

Cook D, Zendejas B, Hamstra S, Hatala R, Brydges R . What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment. Adv Health Sci Educ Theory Pract. 2013; 19(2):233-50. DOI: 10.1007/s10459-013-9458-4. View

14.

Cook D, Triola M . Virtual patients: a critical literature review and proposed next steps. Med Educ. 2009; 43(4):303-11. DOI: 10.1111/j.1365-2923.2008.03286.x. View

15.

Napier F, Davies R, Baldock C, Stevens H, Lockey A, Bullock I . Validation for a scoring system of the ALS cardiac arrest simulation test (CASTest). Resuscitation. 2009; 80(9):1034-8. DOI: 10.1016/j.resuscitation.2009.04.043. View

16.

Cook D, Erwin P, Triola M . Computerized virtual patients in health professions education: a systematic review and meta-analysis. Acad Med. 2010; 85(10):1589-602. DOI: 10.1097/ACM.0b013e3181edfe13. View

17.

Norcini J, Anderson B, Bollela V, Burch V, Costa M, Duvivier R . Criteria for good assessment: consensus statement and recommendations from the Ottawa 2010 Conference. Med Teach. 2011; 33(3):206-14. DOI: 10.3109/0142159X.2011.551559. View

18.

McGaghie W, Issenberg S, Cohen E, Barsuk J, Wayne D . Does simulation-based medical education with deliberate practice yield better results than traditional clinical education? A meta-analytic comparative review of the evidence. Acad Med. 2011; 86(6):706-11. PMC: 3102783. DOI: 10.1097/ACM.0b013e318217e119. View