Reliability Analysis of the Objective Structured Clinical Examination Using Generalizability Theory

Overview

Journal Med Educ Online

Specialty Medical Education

Date 2016 Aug 21

PMID 27543188

Citations 9

Authors

Juan Andres Trejo-Mejia

Melchor Sanchez-Mendiola

Ignacio Mendez-Ramirez

Adrian Martinez-Gonzalez

Affiliations

Soon will be listed here.

Abstract

Background: The objective structured clinical examination (OSCE) is a widely used method for assessing clinical competence in health sciences education. Studies using this method have shown evidence of validity and reliability. There are no published studies of OSCE reliability measurement with generalizability theory (G-theory) in Latin America. The aims of this study were to assess the reliability of an OSCE in medical students using G-theory and explore its usefulness for quality improvement.

Methods: An observational cross-sectional study was conducted at National Autonomous University of Mexico (UNAM) Faculty of Medicine in Mexico City. A total of 278 fifth-year medical students were assessed with an 18-station OSCE in a summative end-of-career final examination. There were four exam versions. G-theory with a crossover random effects design was used to identify the main sources of variance. Examiners, standardized patients, and cases were considered as a single facet of analysis.

Results: The exam was applied to 278 medical students. The OSCE had a generalizability coefficient of 0.93. The major components of variance were stations, students, and residual error. The sites and the versions of the tests had minimum variance.

Conclusions: Our study achieved a G coefficient similar to that found in other reports, which is acceptable for summative tests. G-theory allows the estimation of the magnitude of multiple sources of error and helps decision makers to determine the number of stations, test versions, and examiners needed to obtain reliable measurements.

Citing Articles

Measuring and correcting staff variability in large-scale OSCEs.

Haviari S, de Tymowski C, Burnichon N, Lemogne C, Flamant M, Ruszniewski P BMC Med Educ. 2024; 24(1):817.

PMID: 39075511 PMC: 11288128. DOI: 10.1186/s12909-024-05803-6.

Evaluation of Consecutive Guided Training to Improve Interrater Agreement in Identifying Elements of Situation Awareness in Objective Structured Clinical Examination Assessments.

Fischer M, Schijven M, Kennedy K, Durning S, Kropmans T MedEdPublish (2016). 2024; 10:106.

PMID: 38486551 PMC: 10939566. DOI: 10.15694/mep.2021.000106.1.

Feasibility and reliability of the pandemic-adapted online-onsite hybrid graduation OSCE in Japan.

Hara S, Ohta K, Aono D, Tamai T, Kurachi M, Sugimori K Adv Health Sci Educ Theory Pract. 2023; 29(3):949-965.

PMID: 37851159 PMC: 11208196. DOI: 10.1007/s10459-023-10290-3.

Effectiveness of multi-professional educational interventions to train Comprehensive Geriatric Assessment (CGA) - a Systematic Review.

Lindner-Rabl S, Singler K, Polidori M, Herzog C, Antoniadou E, Seinost G Int J Integr Care. 2023; 23(3):9.

PMID: 37601034 PMC: 10437136. DOI: 10.5334/ijic.7549.

A Tool to Assess Competence in Critical Care Ultrasound Based on Entrustable Professional Activities.

Israel H, Slade M, Gielissen K, Liu R, Pisani M, Chichra A ATS Sch. 2023; 4(1):61-75.

PMID: 37089679 PMC: 10117444. DOI: 10.34197/ats-scholar.2022-0063OC.

References

. PSYCHOLOGY. Estimating the reproducibility of psychological science. Science. 2015; 349(6251):aac4716. DOI: 10.1126/science.aac4716. View

Hull A, Hodder S, Berger B, Ginsberg D, Lindheim N, Quan J . Validity of three clinical performance assessments of internal medicine clerks. Acad Med. 1995; 70(6):517-22. DOI: 10.1097/00001888-199506000-00013. View

Baig L, Violato C . Temporal stability of objective structured clinical exams: a longitudinal study employing item response theory. BMC Med Educ. 2012; 12:121. PMC: 3552978. DOI: 10.1186/1472-6920-12-121. View

Harden R, Stevenson M, Downie W, WILSON G . Assessment of clinical competence using objective structured examination. Br Med J. 1975; 1(5955):447-51. PMC: 1672423. DOI: 10.1136/bmj.1.5955.447. View

Wilkinson T, Newble D, Wilson P, Carter J, Helms R . Development of a three-centre simultaneous objective structured clinical examination. Med Educ. 2000; 34(10):798-807. DOI: 10.1046/j.1365-2923.2000.00669.x. View

Newble D, Swanson D . Psychometric characteristics of the objective structured clinical examination. Med Educ. 1988; 22(4):325-34. DOI: 10.1111/j.1365-2923.1988.tb00761.x. View

Auewarakul C, Downing S, Jaturatamrong U, Praditsuwan R . Sources of validity evidence for an internal medicine student evaluation system: an evaluative study of assessment methods. Med Educ. 2005; 39(3):276-83. DOI: 10.1111/j.1365-2929.2005.02090.x. View

Hatala R, Marr S, Cuncic C, Bacchus C . Modification of an OSCE format to enhance patient continuity in a high-stakes assessment of clinical performance. BMC Med Educ. 2011; 11:23. PMC: 3121725. DOI: 10.1186/1472-6920-11-23. View

Brannick M, Erol-Korkmaz H, Prewett M . A systematic review of the reliability of objective structured clinical examination scores. Med Educ. 2011; 45(12):1181-9. DOI: 10.1111/j.1365-2923.2011.04075.x. View

10.

Bloch R, Norman G . Generalizability theory for the perplexed: a practical introduction and guide: AMEE Guide No. 68. Med Teach. 2012; 34(11):960-92. DOI: 10.3109/0142159X.2012.703791. View

11.

Vallevand A, Violato C . A predictive and construct validity study of a high-stakes objective clinical examination for assessing the clinical competence of international medical graduates. Teach Learn Med. 2012; 24(2):168-76. DOI: 10.1080/10401334.2012.664988. View

12.

Boulet J, McKinley D, Whelan G, Hambleton R . Quality assurance methods for performance-based assessments. Adv Health Sci Educ Theory Pract. 2003; 8(1):27-47. DOI: 10.1023/a:1022639521218. View

13.

Downing S . Reliability: on the reproducibility of assessment data. Med Educ. 2004; 38(9):1006-12. DOI: 10.1111/j.1365-2929.2004.01932.x. View

14.

van der Vleuten C . Validity of final examinations in undergraduate medical training. BMJ. 2000; 321(7270):1217-9. PMC: 1118966. DOI: 10.1136/bmj.321.7270.1217. View

15.

Swanson D, van der Vleuten C . Assessment of clinical skills with standardized patients: state of the art revisited. Teach Learn Med. 2013; 25 Suppl 1:S17-25. DOI: 10.1080/10401334.2013.842916. View

16.

Sanchez-Mendiola M, Durante-Montiel I, Morales-Lopez S, Lozano-Sanchez R, Martinez-Gonzalez A, Graue Wiechers E . [The 2010 curriculum of the faculty of medicine at the National University of Mexico]. Gac Med Mex. 2011; 147(2):152-8. View

17.

Trejo Mejia J, Martinez Gonzalez A, Mendez Ramirez I, Morales Lopez S, Ruiz Perez L, Sanchez Mendiola M . [Clinical competence evaluation using the Objective Structured Clinical Examination (OSCE) in medical internship at UNAM]. Gac Med Mex. 2014; 150(1):8-17. View

18.

Donnon T, Oddone Paolucci E . A generalizability study of the medical judgment vignettes interview to assess students' noncognitive attributes for medical school. BMC Med Educ. 2008; 8:58. PMC: 2625344. DOI: 10.1186/1472-6920-8-58. View

19.

Lawson D . Applying generalizability theory to high-stakes objective structured clinical examinations in a naturalistic environment. J Manipulative Physiol Ther. 2006; 29(6):463-7. DOI: 10.1016/j.jmpt.2006.06.009. View

20.

Downing S . Validity: on meaningful interpretation of assessment data. Med Educ. 2003; 37(9):830-7. DOI: 10.1046/j.1365-2923.2003.01594.x. View