Deep Learning Analysis of Surgical Video Recordings to Assess Nontechnical Skills

Overview

Journal JAMA Netw Open

Publisher American Medical Association

Specialty General Medicine

Date 2024 Jul 31

PMID 39083274

Authors

Rayan Ebnali Harari

Roger D Dias

Lauren R Kennedy-Metz

Giovanna Varni

Matthew Gombolay

Steven Yule

Eduardo Salas

Marco A Zenati

Affiliations

Soon will be listed here.

Abstract

Importance: Assessing nontechnical skills in operating rooms (ORs) is crucial for enhancing surgical performance and patient safety. However, automated and real-time evaluation of these skills remains challenging.

Objective: To explore the feasibility of using motion features extracted from surgical video recordings to automatically assess nontechnical skills during cardiac surgical procedures.

Design, Setting, And Participants: This cross-sectional study used video recordings of cardiac surgical procedures at a tertiary academic US hospital collected from January 2021 through May 2022. The OpenPose library was used to analyze videos to extract body pose estimations of team members and compute various team motion features. The Non-Technical Skills for Surgeons (NOTSS) assessment tool was employed for rating the OR team's nontechnical skills by 3 expert raters.

Main Outcomes And Measures: NOTSS overall score, with motion features extracted from surgical videos as measures.

Results: A total of 30 complete cardiac surgery procedures were included: 26 (86.6%) were on-pump coronary artery bypass graft procedures and 4 (13.4%) were aortic valve replacement or repair procedures. All patients were male, and the mean (SD) age was 72 (6.3) years. All surgical teams were composed of 4 key roles (attending surgeon, attending anesthesiologist, primary perfusionist, and scrub nurse) with additional supporting roles. NOTSS scores correlated significantly with trajectory (r = 0.51, P = .005), acceleration (r = 0.48, P = .008), and entropy (r = -0.52, P = .004) of team displacement. Multiple linear regression, adjusted for patient factors, showed average team trajectory (adjusted R2 = 0.335; coefficient, 10.51 [95% CI, 8.81-12.21]; P = .004) and team displacement entropy (adjusted R2 = 0.304; coefficient, -12.64 [95% CI, -20.54 to -4.74]; P = .003) were associated with NOTSS scores.

Conclusions And Relevance: This study suggests a significant link between OR team movements and nontechnical skills ratings by NOTSS during cardiac surgical procedures, suggesting automated surgical video analysis could enhance nontechnical skills assessment. Further investigation across different hospitals and specialties is necessary to validate these findings.

References

Yule S, Janda A, Likosky D . Surgical Sabermetrics: Applying Athletics Data Science to Enhance Operative Performance. Ann Surg Open. 2021; 2(2):e054. PMC: 8221711. DOI: 10.1097/AS9.0000000000000054. View

Petrosoniak A, Almeida R, Pozzobon L, Hicks C, Fan M, White K . Tracking workflow during high-stakes resuscitation: the application of a novel clinician movement tracing tool during in situ trauma simulation. BMJ Simul Technol Enhanc Learn. 2022; 5(2):78-84. PMC: 8936949. DOI: 10.1136/bmjstel-2017-000300. View

Kurths J, Voss A, Saparin P, Witt A, Kleiner H, Wessel N . Quantitative analysis of heart rate variability. Chaos. 1995; 5(1):88-94. DOI: 10.1063/1.166090. View

Yule S, Gupta A, Gazarian D, Geraghty A, Smink D, Beard J . Construct and criterion validity testing of the Non-Technical Skills for Surgeons (NOTSS) behaviour assessment tool using videos of simulated operations. Br J Surg. 2018; 105(6):719-727. DOI: 10.1002/bjs.10779. View

Jung J, Yule S, Boet S, Szasz P, Schulthess P, Grantcharov T . Nontechnical Skill Assessment of the Collective Surgical Team Using the Non-Technical Skills for Surgeons (NOTSS) System. Ann Surg. 2019; 272(6):1158-1163. DOI: 10.1097/SLA.0000000000003250. View

Barris S, Button C . A review of vision-based motion analysis in sport. Sports Med. 2008; 38(12):1025-43. DOI: 10.2165/00007256-200838120-00006. View

Cao Z, Hidalgo G, Simon T, Wei S, Sheikh Y . OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields. IEEE Trans Pattern Anal Mach Intell. 2019; 43(1):172-186. DOI: 10.1109/TPAMI.2019.2929257. View

Nicksa G, Anderson C, Fidler R, Stewart L . Innovative approach using interprofessional simulation to educate surgical residents in technical and nontechnical skills in high-risk clinical scenarios. JAMA Surg. 2015; 150(3):201-7. DOI: 10.1001/jamasurg.2014.2235. View

Cronin N, Walker J, Tucker C, Nicholson G, Cooke M, Merlino S . Feasibility of OpenPose markerless motion analysis in a real athletics competition. Front Sports Act Living. 2024; 5:1298003. PMC: 10796501. DOI: 10.3389/fspor.2023.1298003. View

10.

Leuschner S, Leuschner M, Kropf S, Niederbichler A . Non-technical skills training in the operating theatre: A meta-analysis of patient outcomes. Surgeon. 2018; 17(4):233-243. DOI: 10.1016/j.surge.2018.07.001. View

11.

Sharma B, Mishra A, Aggarwal R, Grantcharov T . Non-technical skills assessment in surgery. Surg Oncol. 2010; 20(3):169-77. DOI: 10.1016/j.suronc.2010.10.001. View

12.

Tolg B, Lorenz J . An analysis of movement patterns in mass casualty incident simulations. Adv Simul (Lond). 2020; 5:27. PMC: 7547474. DOI: 10.1186/s41077-020-00147-9. View

13.

Fletcher G, Flin R, McGeorge P, Glavin R, Maran N, Patey R . Anaesthetists' Non-Technical Skills (ANTS): evaluation of a behavioural marker system. Br J Anaesth. 2003; 90(5):580-8. DOI: 10.1093/bja/aeg112. View

14.

Dias R, Kennedy-Metz L, Yule S, Gombolay M, Zenati M . Assessing Team Situational Awareness in the Operating Room via Computer Vision. IEEE Conf Cogn Comput Asp Situat Manag. 2022; 2022:94-96. PMC: 9386571. DOI: 10.1109/cogsima54611.2022.9830664. View

15.

Zheng B, Swanstrom L, Mackenzie C . A laboratory study on anticipatory movement in laparoscopic surgery: a behavioral indicator for team collaboration. Surg Endosc. 2006; 21(6):935-40. DOI: 10.1007/s00464-006-9090-y. View

16.

Howie E, Ambler O, Gunn E, Dias R, Wigmore S, Skipworth R . Surgical Sabermetrics: A Scoping Review of Technology-enhanced Assessment of Nontechnical Skills in the Operating Room. Ann Surg. 2024; 279(6):973-984. PMC: 11086675. DOI: 10.1097/SLA.0000000000006211. View

17.

Yule S, Rowley D, Flin R, Maran N, Youngson G, Duncan J . Experience matters: comparing novice and expert ratings of non-technical skills using the NOTSS system. ANZ J Surg. 2009; 79(3):154-60. DOI: 10.1111/j.1445-2197.2008.04833.x. View

18.

Lee A, Finstad A, Gawad N, Boet S, Raiche I, Balaa F . Nontechnical Skills (NTS) in the Undergraduate Surgical and Anesthesiology Curricula: Are We Adequately Preparing Medical Students?. J Surg Educ. 2020; 78(2):502-511. DOI: 10.1016/j.jsurg.2020.08.001. View

19.

. Correction: Competing with lower level opponents decreases intra-team movement synchronization and time-motion demands during pre-season soccer matches. PLoS One. 2015; 10(3):e0120461. PMC: 4361456. DOI: 10.1371/journal.pone.0120461. View

20.

Nakano N, Sakura T, Ueda K, Omura L, Kimura A, Iino Y . Evaluation of 3D Markerless Motion Capture Accuracy Using OpenPose With Multiple Video Cameras. Front Sports Act Living. 2020; 2:50. PMC: 7739760. DOI: 10.3389/fspor.2020.00050. View