A Systematic Review of Annotation for Surgical Process Model Analysis in Minimally Invasive Surgery Based on Video

Overview

Journal Surg Endosc

Publisher Springer

Specialties Gastroenterology
General Surgery
Radiology

Date 2023 May 8

PMID 37157035

Authors

Krystel Nyangoh Timoh

Arnaud Huaulme

Kevin Cleary

Myra A Zaheer

Vincent Lavoue

Dan Donoho

Pierre Jannin

Affiliations

Soon will be listed here.

Abstract

Background: Annotated data are foundational to applications of supervised machine learning. However, there seems to be a lack of common language used in the field of surgical data science. The aim of this study is to review the process of annotation and semantics used in the creation of SPM for minimally invasive surgery videos.

Methods: For this systematic review, we reviewed articles indexed in the MEDLINE database from January 2000 until March 2022. We selected articles using surgical video annotations to describe a surgical process model in the field of minimally invasive surgery. We excluded studies focusing on instrument detection or recognition of anatomical areas only. The risk of bias was evaluated with the Newcastle Ottawa Quality assessment tool. Data from the studies were visually presented in table using the SPIDER tool.

Results: Of the 2806 articles identified, 34 were selected for review. Twenty-two were in the field of digestive surgery, six in ophthalmologic surgery only, one in neurosurgery, three in gynecologic surgery, and two in mixed fields. Thirty-one studies (88.2%) were dedicated to phase, step, or action recognition and mainly relied on a very simple formalization (29, 85.2%). Clinical information in the datasets was lacking for studies using available public datasets. The process of annotation for surgical process model was lacking and poorly described, and description of the surgical procedures was highly variable between studies.

Conclusion: Surgical video annotation lacks a rigorous and reproducible framework. This leads to difficulties in sharing videos between institutions and hospitals because of the different languages used. There is a need to develop and use common ontology to improve libraries of annotated surgical videos.

Citing Articles

LapEx: A new multimodal dataset for context recognition and practice assessment in laparoscopic surgery.

Derathe A, Reche F, Guy S, Charriere K, Trilling B, Jannin P Sci Data. 2025; 12(1):342.

PMID: 40011540 PMC: 11865446. DOI: 10.1038/s41597-025-04588-7.

Using artificial intelligence to evaluate adherence to best practices in one anastomosis gastric bypass: first steps in a real-world setting.

Dayan D, Nizri E, Keidar A Surg Endosc. 2025; 39(3):1945-1951.

PMID: 39870830 PMC: 11870938. DOI: 10.1007/s00464-025-11556-0.

Transferable situation recognition system for scenario-independent context-aware surgical assistance systems: a proof of concept.

Junger D, Kucherer C, Hirt B, Burgert O Int J Comput Assist Radiol Surg. 2024; .

PMID: 39604547 DOI: 10.1007/s11548-024-03283-z.

NeurostimML: a machine learning model for predicting neurostimulation-induced tissue damage.

Li Y, Frederick R, George D, Cogan S, Pancrazio J, Bleris L J Neural Eng. 2024; 21(3).

PMID: 38885676 PMC: 11641559. DOI: 10.1088/1741-2552/ad593e.

Implementation of artificial intelligence-based computer vision model in laparoscopic appendectomy: validation, reliability, and clinical correlation.

Dayan D, Dvir N, Agbariya H, Nizri E Surg Endosc. 2024; 38(6):3310-3319.

PMID: 38664295 DOI: 10.1007/s00464-024-10847-2.

References

Huaulme A, Jannin P, Reche F, Faucheron J, Moreau-Gaudry A, Voros S . Offline identification of surgical deviations in laparoscopic rectopexy. Artif Intell Med. 2020; 104:101837. DOI: 10.1016/j.artmed.2020.101837. View

Gholinejad M, Loeve A, Dankelman J . Surgical process modelling strategies: which method to choose for determining workflow?. Minim Invasive Ther Allied Technol. 2019; 28(2):91-104. DOI: 10.1080/13645706.2019.1591457. View

Kitaguchi D, Takeshita N, Matsuzaki H, Takano H, Owada Y, Enomoto T . Real-time automatic surgical phase recognition in laparoscopic sigmoidectomy using the convolutional neural network-based deep learning approach. Surg Endosc. 2019; 34(11):4924-4931. DOI: 10.1007/s00464-019-07281-0. View

Grenon P, Smith B, Goldberg L . Biodynamic ontology: applying BFO in the biomedical domain. Stud Health Technol Inform. 2005; 102:20-38. View

Riffaud L, Neumuth T, Morandi X, Trantakis C, Meixensberger J, Burgert O . Recording of surgical processes: a study comparing senior and junior neurosurgeons during lumbar disc herniation surgery. Neurosurgery. 2010; 67(2 Suppl Operative):325-32. DOI: 10.1227/NEU.0b013e3181f741d7. View

Cheng K, You J, Wu S, Chen Z, Zhou Z, Guan J . Artificial intelligence-based automated laparoscopic cholecystectomy surgical phase recognition and analysis. Surg Endosc. 2021; 36(5):3160-3168. DOI: 10.1007/s00464-021-08619-3. View

Rosse C, Mejino Jr J . A reference ontology for biomedical informatics: the Foundational Model of Anatomy. J Biomed Inform. 2004; 36(6):478-500. DOI: 10.1016/j.jbi.2003.11.007. View

Meeuwsen F, van Luyn F, Blikkendaal M, Jansen F, van den Dobbelsteen J . Surgical phase modelling in minimal invasive surgery. Surg Endosc. 2018; 33(5):1426-1432. PMC: 6484813. DOI: 10.1007/s00464-018-6417-4. View

Kitaguchi D, Takeshita N, Matsuzaki H, Oda T, Watanabe M, Mori K . Automated laparoscopic colorectal surgery workflow recognition using artificial intelligence: Experimental research. Int J Surg. 2020; 79:88-94. DOI: 10.1016/j.ijsu.2020.05.015. View

10.

Dergachyova O, Bouget D, Huaulme A, Morandi X, Jannin P . Automatic data-driven real-time segmentation and recognition of surgical workflow. Int J Comput Assist Radiol Surg. 2016; 11(6):1081-9. DOI: 10.1007/s11548-016-1371-x. View

11.

Guerin S, Huaulme A, Lavoue V, Jannin P, Nyangoh Timoh K . Review of automated performance metrics to assess surgical technical skills in robot-assisted laparoscopy. Surg Endosc. 2021; 36(2):853-870. DOI: 10.1007/s00464-021-08792-5. View

12.

Meireles O, Rosman G, Altieri M, Carin L, Hager G, Madani A . SAGES consensus recommendations on an annotation framework for surgical video. Surg Endosc. 2021; 35(9):4918-4929. DOI: 10.1007/s00464-021-08578-9. View

13.

Quellec G, Lamard M, Cochener B, Cazuguel G . Real-time segmentation and recognition of surgical tasks in cataract surgery videos. IEEE Trans Med Imaging. 2014; 33(12):2352-60. DOI: 10.1109/TMI.2014.2340473. View

14.

Ma R, Lee R, Nguyen J, Cowan A, Haque T, You J . Tailored Feedback Based on Clinically Relevant Performance Metrics Expedites the Acquisition of Robotic Suturing Skills-An Unblinded Pilot Randomized Controlled Trial. J Urol. 2022; 208(2):414-424. DOI: 10.1097/JU.0000000000002691. View

15.

Guedon A, Meij S, Osman K, Kloosterman H, van Stralen K, Grimbergen M . Deep learning for surgical phase recognition using endoscopic videos. Surg Endosc. 2020; 35(11):6150-6157. DOI: 10.1007/s00464-020-08110-5. View

16.

Mascagni P, Alapatt D, Urade T, Vardazaryan A, Mutter D, Marescaux J . A Computer Vision Platform to Automatically Locate Critical Events in Surgical Videos: Documenting Safety in Laparoscopic Cholecystectomy. Ann Surg. 2021; 274(1):e93-e95. DOI: 10.1097/SLA.0000000000004736. View

17.

Zhang B, Ghanem A, Simes A, Choi H, Yoo A . Surgical workflow recognition with 3DCNN for Sleeve Gastrectomy. Int J Comput Assist Radiol Surg. 2021; 16(11):2029-2036. PMC: 8589754. DOI: 10.1007/s11548-021-02473-3. View

18.

Maier-Hein L, Vedula S, Speidel S, Navab N, Kikinis R, Park A . Surgical data science for next-generation interventions. Nat Biomed Eng. 2019; 1(9):691-696. DOI: 10.1038/s41551-017-0132-7. View

19.

Twinanda A, Shehata S, Mutter D, Marescaux J, De Mathelin M, Padoy N . EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos. IEEE Trans Med Imaging. 2016; 36(1):86-97. DOI: 10.1109/TMI.2016.2593957. View

20.

Katic D, Schuck J, Wekerle A, Kenngott H, Muller-Stich B, Dillmann R . Bridging the gap between formal and experience-based knowledge for context-aware laparoscopy. Int J Comput Assist Radiol Surg. 2016; 11(6):881-8. DOI: 10.1007/s11548-016-1379-2. View