Effect of Rater Training on the Reliability of Technical Skill Assessments: a Randomized Controlled Trial

Overview

Journal Can J Surg

Specialty General Surgery

Date 2018 Sep 29

PMID 30265636

Citations 8

Authors

Reagan L Robertson

Ashley Vergis

Lawrence M Gillman

Jason Park

Affiliations

Soon will be listed here.

Abstract

Background: Rater training improves the reliability of observational assessment tools but has not been well studied for technical skills. This study assessed whether rater training could improve the reliability of technical skill assessment.

Methods: Academic and community surgeons in Royal College of Physicians and Surgeons of Canada surgical subspecialties were randomly allocated to either rater training (7-minute video incorporating frame-of-reference training elements) or no training. Participants then assessed trainees performing a suturing and knot-tying task using 3 assessment tools: a visual analogue scale, a task-specific checklist and a modified version of the Objective Structured Assessment of Technical Skill global rating scale (GRS). We measured interrater reliability (IRR) using intraclass correlation type 2.

Results: There were 24 surgeons in the training group and 23 in the no-training group. Mean assessment tool scores were not significantly different between the 2 groups. The training group had higher IRR than the no-training group on the visual analogue scale (0.71 v. 0.46), task-specific checklist (0.46 v. 0.33) and GRS (0.71 v. 0.61). However, confidence intervals were wide and overlapping for all 3 tools.

Conclusion: For education purposes, the reliability of the visual analogue scale and GRS would be considered "good" for the training group but "moderate" for the no-training group. However, a significant difference in IRR was not shown, and reliability remained below the desired level of 0.8 for high-stakes testing. Training did not significantly improve assessment tool reliability. Although rater training may represent a way to improve reliability, further study is needed to determine effective training methods.

Citing Articles

Trainee anaesthetist self-assessment using an entrustment scale in workplace-based assessment.

Castanelli D, Woods J, Chander A, Weller J Anaesth Intensive Care. 2024; 52(4):241-249.

PMID: 38649296 PMC: 11290023. DOI: 10.1177/0310057X241234676.

Faculty Perceptions of Frame of Reference Training to Improve Workplace-Based Assessment.

Kogan J, Conforti L, Holmboe E J Grad Med Educ. 2023; 15(1):81-91.

PMID: 36817545 PMC: 9934818. DOI: 10.4300/JGME-D-22-00287.1.

Interrater reliability in the assessment of physiotherapy students.

Gittinger F, Lemos M, Neumann J, Forster J, Dohmen D, Berke B BMC Med Educ. 2022; 22(1):186.

PMID: 35296313 PMC: 8928589. DOI: 10.1186/s12909-022-03231-y.

How can surgical skills in laparoscopic colon surgery be objectively assessed?-a scoping review.

Haug T, Worm Orntoft M, Miskovic D, Iversen L, Johnsen S, Madsen A Surg Endosc. 2021; 36(3):1761-1774.

PMID: 34873653 PMC: 8847271. DOI: 10.1007/s00464-021-08914-z.

Assessment of competence in local anaesthetic thoracoscopy: development and validity investigation of a new assessment tool.

Nayahangan L, Svendsen M, Bodtger U, Rahman N, Maskell N, Sidhu J J Thorac Dis. 2021; 13(7):3998-4007.

PMID: 34422330 PMC: 8339737. DOI: 10.21037/jtd-20-3560.

References

Figert P, Park A, Witzke D, Schwartz R . Transfer of training in acquiring laparoscopic skills. J Am Coll Surg. 2001; 193(5):533-7. DOI: 10.1016/s1072-7515(01)01069-9. View

Rogers D, Regehr G, MacDonald J . A role for error training in surgical technical skill instruction and evaluation. Am J Surg. 2002; 183(3):242-5. DOI: 10.1016/s0002-9610(02)00798-5. View

Wanzel K, Ward M, Reznick R . Teaching the surgical craft: From selection to certification. Curr Probl Surg. 2002; 39(6):573-659. DOI: 10.1067/mog.2002.123481. View

Holmboe E, Huot S, Chung J, Norcini J, Hawkins R . Construct validity of the miniclinical evaluation exercise (miniCEX). Acad Med. 2003; 78(8):826-30. DOI: 10.1097/00001888-200308000-00018. View

Noel G, Herbers Jr J, Caplow M, Cooper G, Pangaro L, Harvey J . How well do internal medicine faculty members evaluate the clinical skills of residents?. Ann Intern Med. 1992; 117(9):757-65. DOI: 10.7326/0003-4819-117-9-757. View

Muller M, Dragicevic A . Standardized rater training for the Hamilton Depression Rating Scale (HAMD-17) in psychiatric novices. J Affect Disord. 2003; 77(1):65-9. DOI: 10.1016/s0165-0327(02)00097-6. View

Sidhu R, Grober E, Musselman L, Reznick R . Assessing competency in surgery: where to begin?. Surgery. 2003; 135(1):6-20. DOI: 10.1016/s0039-6060(03)00154-5. View

Kroboth F, Hanusa B, Parker S, Coulehan J, Kapoor W, Brown F . The inter-rater reliability and internal consistency of a clinical evaluation exercise. J Gen Intern Med. 1992; 7(2):174-9. DOI: 10.1007/BF02598008. View

Holmboe E, Hawkins R, Huot S . Effects of training in direct observation of medical residents' clinical competence: a randomized trial. Ann Intern Med. 2004; 140(11):874-81. DOI: 10.7326/0003-4819-140-11-200406010-00008. View

10.

Cusick A, Vasquez M, Knowles L, Wallen M . Effect of rater training on reliability of Melbourne Assessment of Unilateral Upper Limb Function scores. Dev Med Child Neurol. 2005; 47(1):39-45. DOI: 10.1017/s0012162205000071. View

11.

Vassiliou M, Feldman L, Andrew C, Bergman S, Leffondre K, Stanbridge D . A global assessment tool for evaluation of intraoperative laparoscopic skills. Am J Surg. 2005; 190(1):107-13. DOI: 10.1016/j.amjsurg.2005.04.004. View

12.

Evans L, Morse J, Hamann C, Osborne M, Lin Z, DOnofrio G . The development of an independent rater system to assess residents' competence in invasive procedures. Acad Med. 2009; 84(8):1135-43. DOI: 10.1097/ACM.0b013e3181acec7c. View

13.

Vassiliou M, Kaneva P, Poulose B, Dunkin B, Marks J, Sadik R . Global Assessment of Gastrointestinal Endoscopic Skills (GAGES): a valid measurement tool for technical skills in flexible endoscopy. Surg Endosc. 2010; 24(8):1834-41. DOI: 10.1007/s00464-010-0882-8. View

14.

van Hove P, Tuijthof G, Verdaasdonk E, Stassen L, Dankelman J . Objective assessment of technical surgical skills. Br J Surg. 2010; 97(7):972-87. DOI: 10.1002/bjs.7115. View

15.

Crossingham G, Sice P, Roberts M, Lam W, Gale T . Development of workplace-based assessments of non-technical skills in anaesthesia. Anaesthesia. 2012; 67(2):158-64. DOI: 10.1111/j.1365-2044.2011.06977.x. View

16.

Dudek N, Marks M, Wood T, Dojeiji S, Bandiera G, Hatala R . Quality evaluation reports: Can a faculty development program make a difference?. Med Teach. 2012; 34(11):e725-31. DOI: 10.3109/0142159X.2012.689444. View

17.

Feldman M, Lazzara E, Vanderbilt A, DiazGranados D . Rater training to support high-stakes simulation-based assessments. J Contin Educ Health Prof. 2013; 32(4):279-86. PMC: 3646087. DOI: 10.1002/chp.21156. View

18.

Dudek N, Marks M, Bandiera G, White J, Wood T . Quality in-training evaluation reports--does feedback drive faculty performance?. Acad Med. 2013; 88(8):1129-34. DOI: 10.1097/ACM.0b013e318299394c. View

19.

Spanager L, Beier-Holgersen R, Dieckmann P, Konge L, Rosenberg J, Oestergaard D . Reliable assessment of general surgeons' non-technical skills based on video-recordings of patient simulated scenarios. Am J Surg. 2013; 206(5):810-7. DOI: 10.1016/j.amjsurg.2013.04.002. View

20.

George B, Teitelbaum E, DaRosa D, Hungness E, Meyerson S, Fryer J . Duration of faculty training needed to ensure reliable or performance ratings. J Surg Educ. 2013; 70(6):703-8. DOI: 10.1016/j.jsurg.2013.06.015. View