Mobile Application to Improve Just-in-Time 2-Way Formative Feedback in Graduate Medical Education

Overview

Journal J Grad Med Educ

Date 2024 Jul 12

PMID 38993301

Authors

Jane Rowat

Girish Mhatre

Manish Suneja

Affiliations

Soon will be listed here.

Abstract

An easy-to-use application to facilitate direct observation and allow for 2-way feedback between residents and faculty is needed. To develop a mobile-based application (app) with the goals of (1) providing just-in-time feedback to residents; (2) improving timeliness of feedback by faculty; and (3) allowing residents to comment on the value of faculty feedback. Fifty-one of 69 (74%) internal medicine (IM) residents and 20 of 25 (80%) IM core faculty participated in the study from July 1, 2020, to December 31, 2021. An iOS app was designed by authors with expertise in medical education and application development to capture entrustable professional activities (EPAs)-based feedback (eg, informed consent) based on direct observation of residents' skills in the workplace. App utilization and narrative feedback characteristics of faculty comments were examined by exporting the data from the database server. The end user satisfaction was examined using a survey instrument. Eighty-seven percent of assessments (117 of 134) initiated were fully completed by residents and faculty. Faculty narrative comments were noted in 97% (114 of 117) of completed assessments and 64% (75 of 117) of residents' feedback to the faculty contained narrative comments. Eighty-three percent (97 of 117) of comments were behaviorally specific and 71% (83 of 117) contained an actionable item. Eighty-six percent (18 of 21) of residents and 90% (9 of 10) of core faculty stated that this application promoted an educational interaction between them. This app facilitates the efficient completion of EPA-based formative assessments and captures bidirectional feedback in the workplace setting.

References

Bohnen J, George B, Williams R, Schuller M, DaRosa D, Torbeck L . The Feasibility of Real-Time Intraoperative Performance Assessment With SIMPL (System for Improving and Measuring Procedural Learning): Early Experience From a Multi-institutional Trial. J Surg Educ. 2016; 73(6):e118-e130. DOI: 10.1016/j.jsurg.2016.08.010. View

Torre D, Simpson D, Elnicki D, Sebastian J, Holmboe E . Feasibility, reliability and user satisfaction with a PDA-based mini-CEX to evaluate the clinical skills of third-year medical students. Teach Learn Med. 2007; 19(3):271-7. DOI: 10.1080/10401330701366622. View

Eaton M, Scully R, Schuller M, Yang A, Smink D, Williams R . Value and Barriers to Use of the SIMPL Tool for Resident Feedback. J Surg Educ. 2019; 76(3):620-627. DOI: 10.1016/j.jsurg.2019.01.012. View

Lomis K, Amiel J, Ryan M, Esposito K, Green M, Stagnaro-Green A . Implementing an Entrustable Professional Activities Framework in Undergraduate Medical Education: Early Lessons From the AAMC Core Entrustable Professional Activities for Entering Residency Pilot. Acad Med. 2017; 92(6):765-770. DOI: 10.1097/ACM.0000000000001543. View

Hicks P, Margolis M, Carraccio C, Clauser B, Donnelly K, Fromme H . A novel workplace-based assessment for competency-based decisions and learner feedback. Med Teach. 2018; 40(11):1143-1150. DOI: 10.1080/0142159X.2018.1461204. View

Hasley P, Arnold R . Summative evaluation on the hospital wards. What do faculty say to learners?. Adv Health Sci Educ Theory Pract. 2008; 14(3):431-9. DOI: 10.1007/s10459-008-9127-1. View

George B, Bohnen J, Williams R, Meyerson S, Schuller M, Clark M . Readiness of US General Surgery Residents for Independent Practice. Ann Surg. 2017; 266(4):582-594. DOI: 10.1097/SLA.0000000000002414. View

Lockyer J, Sargeant J, Richards S, Campbell J, Rivera L . Multisource Feedback and Narrative Comments: Polarity, Specificity, Actionability, and CanMEDS Roles. J Contin Educ Health Prof. 2018; 38(1):32-40. DOI: 10.1097/CEH.0000000000000183. View

Gigante J, Dell M, Sharkey A . Getting beyond "Good job": how to give effective feedback. Pediatrics. 2011; 127(2):205-7. DOI: 10.1542/peds.2010-3351. View

10.

Fitzpatrick R, Paterson N, Watterson J, Seabrook C, Roberts M . Development and implementation of a mobile version of the O-SCORE assessment tool and case log for competency-based assessment in urology residency training: An initial assessment of utilization and acceptance among residents and faculty. Can Urol Assoc J. 2018; 13(2):45-50. PMC: 6363562. DOI: 10.5489/cuaj.5482. View

11.

Cantillon P, Sargeant J . Giving feedback in clinical settings. BMJ. 2008; 337:a1961. DOI: 10.1136/bmj.a1961. View

12.

Young J, Sugarman R, Schwartz J, McClure M, OSullivan P . A mobile app to capture EPA assessment data: Utilizing the consolidated framework for implementation research to identify enablers and barriers to engagement. Perspect Med Educ. 2020; 9(4):210-219. PMC: 7459074. DOI: 10.1007/s40037-020-00587-z. View

13.

Cooney C, Redett 3rd R, Dorafshar A, Zarrabi B, Lifchez S . Integrating the NAS Milestones and handheld technology to improve residency training and assessment. J Surg Educ. 2014; 71(1):39-42. DOI: 10.1016/j.jsurg.2013.09.019. View

14.

CarlLee S, Rowat J, Suneja M . Assessing Entrustable Professional Activities Using an Orientation OSCE: Identifying the Gaps. J Grad Med Educ. 2019; 11(2):214-220. PMC: 6476083. DOI: 10.4300/JGME-D-18-00601.2. View

15.

Van Hell E, Kuks J, Raat A, van Lohuizen M, Cohen-Schotanus J . Instructiveness of feedback during clerkships: influence of supervisor, observation and student initiative. Med Teach. 2008; 31(1):45-50. DOI: 10.1080/01421590802144294. View

16.

Page C, Reid A, Coe C, Carlough M, Rosenbaum D, Beste J . Learnings From the Pilot Implementation of Mobile Medical Milestones Application. J Grad Med Educ. 2016; 8(4):569-575. PMC: 5058591. DOI: 10.4300/JGME-D-15-00550.1. View

17.

Diwersi N, Gass J, Fischer H, Metzger J, Knobe M, Marty A . Surgery goes EPA (Entrustable Professional Activity) - how a strikingly easy to use app revolutionizes assessments of clinical skills in surgical training. BMC Med Educ. 2022; 22(1):559. PMC: 9295378. DOI: 10.1186/s12909-022-03622-1. View

18.

Carraccio C, Englander R, Gilhooly J, Mink R, Hofkosh D, Barone M . Building a Framework of Entrustable Professional Activities, Supported by Competencies and Milestones, to Bridge the Educational Continuum. Acad Med. 2016; 92(3):324-330. DOI: 10.1097/ACM.0000000000001141. View

19.

Holmboe E . Faculty and the observation of trainees' clinical skills: problems and opportunities. Acad Med. 2003; 79(1):16-22. DOI: 10.1097/00001888-200401000-00006. View

20.

Lypson M, Frohna J, Gruppen L, Woolliscroft J . Assessing residents' competencies at baseline: identifying the gaps. Acad Med. 2004; 79(6):564-70. DOI: 10.1097/00001888-200406000-00013. View