Investigation on Performance of Multiple AI-based Auto-contouring Systems in Organs at Risks (OARs) Delineation

Overview

Journal Phys Eng Sci Med

Publisher Springer

Specialty Biomedical Engineering

Date 2024 Sep 2

PMID 39222214

Authors

Young Woo Kim

Simon Biggs

Elizabeth Claridge Mackonis

Affiliations

Soon will be listed here.

Abstract

Manual contouring of organs at risk (OAR) is time-consuming and subject to inter-observer variability. AI-based auto-contouring is proposed as a solution to these problems if it can produce clinically acceptable results. This study investigated the performance of multiple AI-based auto-contouring systems in different OAR segmentations. The auto-contouring was performed using seven different AI-based segmentation systems (Radiotherapy AI, Limbus AI version 1.5 and 1.6, Therapanacea, MIM, Siemens AI-Rad Companion and RadFormation) on a total of 42 clinical cases with varying anatomical sites. Volumetric and surface dice similarity coefficients and maximum Hausdorff distance (HD) between the expert's contours and automated contours were calculated to evaluate their performance. Radiotherapy AI has shown better performance than other software in most tested structures considered in the head and neck, and brain cases. No specific software had shown overall superior performance over other software in lung, breast, pelvis and abdomen cases. Each tested AI system was able to produce comparable contours to the experts' contours of organs at risk which can potentially be used for clinical use. A reduced performance of AI systems in the case of small and complex anatomical structures was found and reported, showing that it is still essential to review each contour produced by AI systems for clinical uses. This study has also demonstrated a method of comparing contouring software options which could be replicated in clinics or used for ongoing quality assurance of purchased systems.

Citing Articles

Automated Organ Segmentation for Radiation Therapy: A Comparative Analysis of AI-Based Tools Versus Manual Contouring in Korean Cancer Patients.

Choi S, Park J, Cho Y, Yang G, Yoon H Cancers (Basel). 2024; 16(21).

PMID: 39518109 PMC: 11544936. DOI: 10.3390/cancers16213670.

References

Nikolov S, Blackwell S, Zverovitch A, Mendes R, Livne M, De Fauw J . Clinically Applicable Segmentation of Head and Neck Anatomy for Radiotherapy: Deep Learning Algorithm Development and Validation Study. J Med Internet Res. 2021; 23(7):e26151. PMC: 8314151. DOI: 10.2196/26151. View

Vaassen F, Hazelaar C, Vaniqui A, Gooding M, van der Heyden B, Canters R . Evaluation of measures for assessing time-saving of automatic organ-at-risk segmentation in radiotherapy. Phys Imaging Radiat Oncol. 2021; 13:1-6. PMC: 7807544. DOI: 10.1016/j.phro.2019.12.001. View

Chen W, Li Y, Dyer B, Feng X, Rao S, Benedict S . Deep learning vs. atlas-based models for fast auto-segmentation of the masticatory muscles on head and neck CT images. Radiat Oncol. 2020; 15(1):176. PMC: 7372849. DOI: 10.1186/s13014-020-01617-0. View

Chen Z, King W, Pearcey R, Kerba M, Mackillop W . The relationship between waiting time for radiotherapy and clinical outcomes: a systematic review of the literature. Radiother Oncol. 2007; 87(1):3-16. DOI: 10.1016/j.radonc.2007.11.016. View

Baroudi H, Brock K, Cao W, Chen X, Chung C, Court L . Automated Contouring and Planning in Radiation Therapy: What Is 'Clinically Acceptable'?. Diagnostics (Basel). 2023; 13(4). PMC: 9955359. DOI: 10.3390/diagnostics13040667. View

Doolan P, Charalambous S, Roussakis Y, Leczynski A, Peratikou M, Benjamin M . A clinical evaluation of the performance of five commercial artificial intelligence contouring systems for radiotherapy. Front Oncol. 2023; 13:1213068. PMC: 10436522. DOI: 10.3389/fonc.2023.1213068. View

Daisne J, Blumhofer A . Atlas-based automatic segmentation of head and neck organs at risk and nodal target volumes: a clinical validation. Radiat Oncol. 2013; 8:154. PMC: 3722083. DOI: 10.1186/1748-717X-8-154. View

Whitley E, Ball J . Statistics review 6: Nonparametric methods. Crit Care. 2002; 6(6):509-13. PMC: 153434. DOI: 10.1186/cc1820. View

Choi M, Choi B, Chung S, Kim N, Chun J, Kim Y . Clinical evaluation of atlas- and deep learning-based automatic segmentation of multiple organs and clinical target volumes for breast cancer. Radiother Oncol. 2020; 153:139-145. DOI: 10.1016/j.radonc.2020.09.045. View

10.

Ng S, Dyer B, Kalpathy-Cramer J, Mohamed A, Awan M, Gunn G . A prospective in silico analysis of interdisciplinary and interobserver spatial variability in post-operative target delineation of high-risk oral cavity cancers: Does physician specialty matter?. Clin Transl Radiat Oncol. 2018; 12:40-46. PMC: 6105928. DOI: 10.1016/j.ctro.2018.07.006. View

11.

AlBadawy E, Saha A, Mazurowski M . Deep learning for segmentation of brain tumors: Impact of cross-institutional training and testing. Med Phys. 2018; 45(3):1150-1158. DOI: 10.1002/mp.12752. View

12.

Zabel W, Conway J, Gladwish A, Skliarenko J, DiDiodato G, Goorts-Matthews L . Clinical Evaluation of Deep Learning and Atlas-Based Auto-Contouring of Bladder and Rectum for Prostate Radiation Therapy. Pract Radiat Oncol. 2020; 11(1):e80-e89. DOI: 10.1016/j.prro.2020.05.013. View

13.

Li X, Tai A, Arthur D, Buchholz T, MacDonald S, Marks L . Variability of target and normal structure delineation for breast cancer radiotherapy: an RTOG Multi-Institutional and Multiobserver Study. Int J Radiat Oncol Biol Phys. 2009; 73(3):944-51. PMC: 2911777. DOI: 10.1016/j.ijrobp.2008.10.034. View

14.

Radici L, Ferrario S, Casanova Borca V, Cante D, Paolini M, Piva C . Implementation of a Commercial Deep Learning-Based Auto Segmentation Software in Radiotherapy: Evaluation of Effectiveness and Impact on Workflow. Life (Basel). 2022; 12(12). PMC: 9782080. DOI: 10.3390/life12122088. View

15.

LeCun Y, Bengio Y, Hinton G . Deep learning. Nature. 2015; 521(7553):436-44. DOI: 10.1038/nature14539. View

16.

Fay M, Proschan M . Wilcoxon-Mann-Whitney or t-test? On assumptions for hypothesis tests and multiple interpretations of decision rules. Stat Surv. 2010; 4:1-39. PMC: 2857732. DOI: 10.1214/09-SS051. View

17.

Ahn S, Yeo A, Kim K, Kim C, Goh Y, Cho S . Comparative clinical evaluation of atlas and deep-learning-based auto-segmentation of organ structures in liver cancer. Radiat Oncol. 2019; 14(1):213. PMC: 6880380. DOI: 10.1186/s13014-019-1392-z. View

18.

Liu P, Sun Y, Zhao X, Yan Y . Deep learning algorithm performance in contouring head and neck organs at risk: a systematic review and single-arm meta-analysis. Biomed Eng Online. 2023; 22(1):104. PMC: 10621161. DOI: 10.1186/s12938-023-01159-y. View

19.

Taha A, Hanbury A . Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool. BMC Med Imaging. 2015; 15:29. PMC: 4533825. DOI: 10.1186/s12880-015-0068-x. View

20.

Rhee D, Anakwenze Akinfenwa C, Rigaud B, Jhingran A, Cardenas C, Zhang L . Automatic contouring QA method using a deep learning-based autocontouring system. J Appl Clin Med Phys. 2022; 23(8):e13647. PMC: 9359039. DOI: 10.1002/acm2.13647. View