Evaluation of the Clinical Efficacy and Trust in AI-Assisted Embryo Ranking: Survey-Based Prospective Study

Overview

Journal J Med Internet Res

Publisher JMIR Publications

Specialty Medical Informatics

Date 2024 Jun 3

PMID 38830209

Authors

Hyung Min Kim

Hyoeun Kang

Chaeyoon Lee

Jong Hyuk Park

Mi Kyung Chung

Miran Kim

Na Young Kim

Hye Jun Lee

Affiliations

Soon will be listed here.

Abstract

Background: Current embryo assessment methods for in vitro fertilization depend on subjective morphological assessments. Recently, artificial intelligence (AI) has emerged as a promising tool for embryo assessment; however, its clinical efficacy and trustworthiness remain unproven. Simulation studies may provide additional evidence, provided that they are meticulously designed to mitigate bias and variance.

Objective: The primary objective of this study was to evaluate the benefits of an AI model for predicting clinical pregnancy through well-designed simulations. The secondary objective was to identify the characteristics of and potential bias in the subgroups of embryologists with varying degrees of experience.

Methods: This simulation study involved a questionnaire-based survey conducted on 61 embryologists with varying levels of experience from 12 in vitro fertilization clinics. The survey was conducted via Google Forms (Google Inc) in three phases: (1) phase 1, an initial assessment (December 23, 2022, to January 22, 2023); (2) phase 2, a validation assessment (March 6, 2023, to April 5, 2023); and (3) phase 3 an AI-guided assessment (March 6, 2023, to April 5, 2023). Inter- and intraobserver assessments and the accuracy of embryo selection from 360 day-5 embryos before and after AI guidance were analyzed for all embryologists and subgroups of senior and junior embryologists.

Results: With AI guidance, the interobserver agreement increased from 0.355 to 0.527 and from 0.440 to 0.524 for junior and senior embryologists, respectively, thus reaching similar levels of agreement. In a test of accurate embryo selection with 90 questions, the numbers of correct responses by the embryologists only, embryologists with AI guidance, and AI only were 34 (38%), 45 (50%), and 59 (66%), respectively. Without AI, the average score (accuracy) of the junior group was 33.516 (37%), while that of the senior group was 35.967 (40%), with P<.001 in the t test. With AI guidance, the average score (accuracy) of the junior group increased to 46.581 (52%), reaching a level similar to that of the senior embryologists of 44.833 (50%), with P=.34. Junior embryologists had a higher level of trust in the AI score.

Conclusions: This study demonstrates the potential benefits of AI in selecting embryos with high chances of pregnancy, particularly for embryologists with 5 years or less of experience, possibly due to their trust in AI. Thus, using AI as an auxiliary tool in clinical practice has the potential to improve embryo assessment and increase the probability of a successful pregnancy.

Citing Articles

Reporting Quality of AI Intervention in Randomized Controlled Trials in Primary Care: Systematic Review and Meta-Epidemiological Study.

Zhong J, Zhu T, Huang Y J Med Internet Res. 2025; 27:e56774.

PMID: 39998876 PMC: 11897677. DOI: 10.2196/56774.

References

Paternot G, Devroe J, Debrock S, DHooghe T, Spiessens C . Intra- and inter-observer analysis in the morphological assessment of early-stage embryos. Reprod Biol Endocrinol. 2009; 7:105. PMC: 2761923. DOI: 10.1186/1477-7827-7-105. View

Wang J, Sauer M . In vitro fertilization (IVF): a review of 3 decades of clinical innovation and technological advancement. Ther Clin Risk Manag. 2008; 2(4):355-64. PMC: 1936357. DOI: 10.2147/tcrm.2006.2.4.355. View

Chow D, Wijesinghe P, Dholakia K, Dunning K . Does artificial intelligence have a role in the IVF clinic?. Reprod Fertil. 2022; 2(3):C29-C34. PMC: 8801019. DOI: 10.1530/RAF-21-0043. View

Kirkegaard K, Agerholm I, Ingerslev H . Time-lapse monitoring as a tool for clinical embryo assessment. Hum Reprod. 2012; 27(5):1277-85. DOI: 10.1093/humrep/des079. View

Tran D, Cooke S, Illingworth P, Gardner D . Deep learning as a predictive tool for fetal heart pregnancy following time-lapse incubation and blastocyst transfer. Hum Reprod. 2019; 34(6):1011-1018. PMC: 6554189. DOI: 10.1093/humrep/dez064. View

Khosravi P, Kazemi E, Zhan Q, Malmsten J, Toschi M, Zisimopoulos P . Deep learning enables robust assessment and selection of human blastocysts after in vitro fertilization. NPJ Digit Med. 2019; 2:21. PMC: 6550169. DOI: 10.1038/s41746-019-0096-y. View

Adolfsson E, Andershed A . Morphology vs morphokinetics: a retrospective comparison of inter-observer and intra-observer agreement between embryologists on blastocysts with known implantation outcome. JBRA Assist Reprod. 2018; 22(3):228-237. PMC: 6106622. DOI: 10.5935/1518-0557.20180042. View

Kim H, Ko T, Kang H, Choi S, Park J, Chung M . Improved prediction of clinical pregnancy using artificial intelligence with enhanced inner cell mass and trophectoderm images. Sci Rep. 2024; 14(1):3240. PMC: 10853203. DOI: 10.1038/s41598-024-52241-x. View

Cornelisse S, Zagers M, Kostova E, Fleischer K, van Wely M, Mastenbroek S . Preimplantation genetic testing for aneuploidies (abnormal number of chromosomes) in in vitro fertilisation. Cochrane Database Syst Rev. 2020; 9:CD005291. PMC: 8094272. DOI: 10.1002/14651858.CD005291.pub3. View

10.

Diakiw S, Hall J, VerMilyea M, Lim A, Quangkananurug W, Chanchamroen S . An artificial intelligence model correlated with morphological and genetic features of blastocyst quality improves ranking of viable embryos. Reprod Biomed Online. 2022; 45(6):1105-1117. DOI: 10.1016/j.rbmo.2022.07.018. View

11.

Sundvall L, Ingerslev H, Breth Knudsen U, Kirkegaard K . Inter- and intra-observer variability of time-lapse annotations. Hum Reprod. 2013; 28(12):3215-21. DOI: 10.1093/humrep/det366. View

12.

VerMilyea M, Hall J, Diakiw S, Johnston A, Nguyen T, Perugini D . Development of an artificial intelligence-based assessment model for prediction of embryo viability using static images captured by optical light microscopy during IVF. Hum Reprod. 2020; 35(4):770-784. PMC: 7192535. DOI: 10.1093/humrep/deaa013. View

13.

Racowsky C, Vernon M, Mayer J, Ball G, Behr B, Pomeroy K . Standardization of grading embryo morphology. J Assist Reprod Genet. 2010; 27(8):437-9. PMC: 2941588. DOI: 10.1007/s10815-010-9443-2. View

14.

Bori L, Meseguer F, Valera M, Galan A, Remohi J, Meseguer M . The higher the score, the better the clinical outcome: retrospective evaluation of automatic embryo grading as a support tool for embryo selection in IVF laboratories. Hum Reprod. 2022; 37(6):1148-1160. DOI: 10.1093/humrep/deac066. View

15.

Chen M, Wei S, Hu J, Yuan J, Liu F . Does time-lapse imaging have favorable results for embryo incubation and selection compared with conventional methods in clinical in vitro fertilization? A meta-analysis and systematic review of randomized controlled trials. PLoS One. 2017; 12(6):e0178720. PMC: 5453598. DOI: 10.1371/journal.pone.0178720. View

16.

Chavez-Badiola A, Flores-Saiffe-Farias A, Mendizabal-Ruiz G, Drakeley A, Cohen J . Embryo Ranking Intelligent Classification Algorithm (ERICA): artificial intelligence clinical assistant predicting embryo ploidy and implantation. Reprod Biomed Online. 2020; 41(4):585-593. DOI: 10.1016/j.rbmo.2020.07.003. View

17.

Storr A, Venetis C, Cooke S, Kilani S, Ledger W . Inter-observer and intra-observer agreement between embryologists during selection of a single Day 5 embryo for transfer: a multicenter study. Hum Reprod. 2016; 32(2):307-314. DOI: 10.1093/humrep/dew330. View

18.

. The Istanbul consensus workshop on embryo assessment: proceedings of an expert meeting. Hum Reprod. 2011; 26(6):1270-83. DOI: 10.1093/humrep/der037. View

19.

Hill M, Richter K, Heitmann R, Graham J, Tucker M, DeCherney A . Trophectoderm grade predicts outcomes of single-blastocyst transfers. Fertil Steril. 2013; 99(5):1283-1289.e1. DOI: 10.1016/j.fertnstert.2012.12.003. View

20.

Bormann C, Thirumalaraju P, Kanakasabapathy M, Kandula H, Souter I, Dimitriadis I . Consistency and objectivity of automated embryo assessments using deep neural networks. Fertil Steril. 2020; 113(4):781-787.e1. PMC: 7583085. DOI: 10.1016/j.fertnstert.2019.12.004. View