Effectiveness of a Focused Educational Intervention on Resident Evaluations from Faculty a Randomized Controlled Trial

Overview

Journal J Gen Intern Med

Publisher Springer

Specialty General Medicine

Date 2001 Aug 25

PMID 11520379

Citations 21

Authors

E S Holmboe

N H Fiebach

L A Galaty

S Huot

Affiliations

Soon will be listed here.

Abstract

Objective: To improve the quality and specificity of written evaluations by faculty attendings of internal medicine residents during inpatient rotations.

Design: Prospective randomized controlled trial.

Setting: Four hospitals: tertiary care university hospital, Veterans' Administration hospital, and two community hospitals.

Participants: Eighty-eight faculty and 157 residents from categorical and primary-care internal medicine residency training programs rotating on inpatient general medicine teams.

Intervention: Focused 20-minute educational session on evaluation and feedback, accompanied by 3 by 5 reminder card and diary, given to faculty at the start of their attending month.

Measurements And Main Results:

Primary Outcomes: 1) number of written comments from faculty specific to unique, preselected dimensions of competence; 2) number of written comments from faculty describing a specific resident behavior or providing a recommendation; and 3) resident Likert-scale ratings of the quantity and effect of feedback received from faculty. Faculty in the intervention group provided more written comments specific to defined dimensions of competence, a median of three comments per evaluation form versus two in the control group, but when adjusted for clustering by faculty, the difference was not statistically significant (P =.09). Regarding feedback, residents in the intervention group rated the quantity significantly higher (P =.04) and were significantly more likely to make changes in clinical management of patients than residents in the control group (P =.04).

Conclusions: A brief, focused educational intervention delivered to faculty prior to the start of a ward rotation appears to have a modest effect on faculty behavior for written evaluations and promoted higher quality feedback given to house staff.

Citing Articles

What/Why/When/Where/How Framework and Faculty Development Workshop to Improve the Utility of Narrative Evaluations for Assessing Internal Medicine Residents.

Sekar D, Ehrenberger K, Dakroub A, Rothenberger S, Grau T, Carter A MedEdPORTAL. 2024; 20:11420.

PMID: 39081631 PMC: 11286767. DOI: 10.15766/mep_2374-8265.11420.

Novel In-Training Evaluation Report in an Internal Medicine Residency Program: Improving the Quality of the Narrative Assessment.

Gutierrez M, Wilson K, Bickford B, Yuhas J, Markert R, Burtson K J Med Educ Curric Dev. 2023; 10:23821205231206058.

PMID: 37822780 PMC: 10563452. DOI: 10.1177/23821205231206058.

The influence of clinical coaching teams on quality of entrustable professional activity assessments.

Singh S, Cheung W, Dewhirst S, Wood T, Landreville J AEM Educ Train. 2023; 7(3):e10879.

PMID: 37361186 PMC: 10290210. DOI: 10.1002/aet2.10879.

Feedback in Medical Education: An Evidence-based Guide to Best Practices from the Council of Residency Directors in Emergency Medicine.

Natesan S, Jordan J, Sheng A, Carmelli G, Barbas B, King A West J Emerg Med. 2023; 24(3):479-494.

PMID: 37278777 PMC: 10284500. DOI: 10.5811/westjem.56544.

The integrative feedback tool: assessing a novel feedback tool among emergency medicine residents.

Gore K, Schiebout J, Peksa G, Hock S, Patwari R, Gottlieb M Clin Exp Emerg Med. 2023; 10(3):306-314.

PMID: 36796780 PMC: 10579731. DOI: 10.15441/ceem.22.395.

References

Johansson J, Skeff K, Stratos G . Clinical teaching improvement: The transportability of the Stanford Faculty Development Program. Med Teach. 2009; 31(8):e377-82. DOI: 10.1080/01421590802638055. View

Ende J . Feedback in clinical medical education. JAMA. 1983; 250(6):777-81. View

Davis J, Inamdar S, Stone R . Interrater agreement and predictive validity of faculty ratings of pediatric residents. J Med Educ. 1986; 61(11):901-5. DOI: 10.1097/00001888-198611000-00006. View

Kalet A, Earp J, Kowlowitz V . How well do faculty evaluate the interviewing skills of medical students?. J Gen Intern Med. 1992; 7(5):499-505. DOI: 10.1007/BF02599452. View

Veet L, Shea J, Ende J . Our continuing interest in manuscripts about education. J Gen Intern Med. 2007; 12(9):583-5. PMC: 1497165. DOI: 10.1046/j.1525-1497.1997.07115.x. View

Thompson W, Lipkin Jr M, Gilbert D, Guzzo R, Roberson L . Evaluating evaluation: assessment of the American Board of Internal Medicine Resident Evaluation Form. J Gen Intern Med. 1990; 5(3):214-7. DOI: 10.1007/BF02600537. View

Holmboe E, Hawkins R . Methods for evaluating the clinical competence of residents in internal medicine: a review. Ann Intern Med. 1998; 129(1):42-8. DOI: 10.7326/0003-4819-129-1-199807010-00011. View

Carline J, Paauw D, Thiede K, Ramsey P . Factors affecting the reliability of ratings of students' clinical skills in a medicine clerkship. J Gen Intern Med. 1992; 7(5):506-10. DOI: 10.1007/BF02599454. View

Norcini Jr J . Standards and reliability in evaluation: when rules of thumb don't apply. Acad Med. 1999; 74(10):1088-90. DOI: 10.1097/00001888-199910000-00010. View

10.

Noel G, Herbers Jr J, Caplow M, Cooper G, Pangaro L, Harvey J . How well do internal medicine faculty members evaluate the clinical skills of residents?. Ann Intern Med. 1992; 117(9):757-65. DOI: 10.7326/0003-4819-117-9-757. View

11.

. Guidelines for evaluating papers on educational interventions. BMJ. 1999; 318(7193):1265-7. PMC: 1115651. View

12.

GRAY J . Global rating scales in residency education. Acad Med. 1996; 71(1 Suppl):S55-63. DOI: 10.1097/00001888-199601000-00043. View

13.

Norcini J, Blank L, Arnold G, Kimball H . The mini-CEX (clinical evaluation exercise): a preliminary investigation. Ann Intern Med. 1995; 123(10):795-9. DOI: 10.7326/0003-4819-123-10-199511150-00008. View

14.

Eisenberg J . Evaluating internists' clinical competence. J Gen Intern Med. 1989; 4(2):139-43. DOI: 10.1007/BF02602356. View

15.

Haber R, Avins A . Do ratings on the American Board of Internal Medicine Resident Evaluation Form detect differences in clinical competence?. J Gen Intern Med. 1994; 9(3):140-5. DOI: 10.1007/BF02600028. View

16.

Turnbull J, Gray J, MacFadyen J . Improving in-training evaluation programs. J Gen Intern Med. 1998; 13(5):317-23. PMC: 1496958. DOI: 10.1046/j.1525-1497.1998.00097.x. View

17.

Maxim B, Dielman T . Dimensionality, internal consistency and interrater reliability of clinical performance ratings. Med Educ. 1987; 21(2):130-7. DOI: 10.1111/j.1365-2923.1987.tb00679.x. View

18.

Kroboth F, Hanusa B, Parker S, Coulehan J, Kapoor W, Brown F . The inter-rater reliability and internal consistency of a clinical evaluation exercise. J Gen Intern Med. 1992; 7(2):174-9. DOI: 10.1007/BF02598008. View

19.

Herbers Jr J, Noel G, Cooper G, Harvey J, Pangaro L, Weaver M . How accurate are faculty evaluations of clinical competence?. J Gen Intern Med. 1989; 4(3):202-8. DOI: 10.1007/BF02599524. View