Interrater and Intrarater Reliability in the Diagnosis and Staging of Endometriosis

Overview

Journal Obstet Gynecol

Specialty Gynecology & Obstetrics

Date 2012 Aug 24

PMID 22914398

Citations 15

Authors

Karen C Schliep

Joseph B Stanford

Zhen Chen

Bo Zhang

Jessie K Dorais

Erica Boiman Johnstone

Ahmad O Hammoud

Michael W Varner

Germaine M Buck Louis

C Matthew Peterson

Affiliations

Soon will be listed here.

Abstract

Objective: To estimate the interrater and intrarater reliability of endometriosis diagnosis and severity of disease among gynecologic surgeons viewing operative digital images.

Methods: The study population comprised a random sample (n=148 [36%]) of women who participated in the Endometriosis: Natural History, Diagnosis and Outcomes study. Four academic expert and four local, specialized expert surgeons reviewed the images, diagnosed the presence or absence of endometriosis for each woman, and rated severity using the revised American Society for Reproductive Medicine (ASRM) criteria. Interrater-level and intrarater-level agreement were calculated for both endometriosis diagnosis and staging.

Results: The interrater reliability for endometriosis diagnosis among the eight surgeons was substantial: Fleiss κ=0.69 (95% confidence interval [CI] 0.64-0.74). Surgeons agreed on revised ASRM endometriosis staging criteria after experienced assessment in a majority of cases (mean 61%, range 52-75%) with moderate interrater reliability: Fleiss κ=0.44 (95% CI 0.41-0.47). The intrarater reliability for experienced assessment compared with computer-assisted revised ASRM staging was almost perfect (mean weighted κ=0.95, range 0.89-0.99).

Conclusion: Substantial reliability was found for revised ASRM endometriosis diagnosis, whereas moderate reliability was observed for staging. Almost perfect reliability was observed for surgeons' rating of disease severity compared with computerized-assisted, checklist-based staging. Findings suggest that reliability in endometriosis diagnosis is not greatly altered by location or composition of surgeons, supporting the conduct of multisite studies or compilation of endometriosis data across clinical centers. Although surgeons appear to be skilled at assessing endometriosis stage intuitively, how staging of disease burden correlates with clinical outcomes remains to be developed.

Citing Articles

Is endometriosis typology a potentially better classification system for assessing risk of female infertility?.

Schliep K, Pollack A, Farland L, Shaaban M, Yan B, Wang J F S Rep. 2025; 5(4):394-401.

PMID: 39781080 PMC: 11705600. DOI: 10.1016/j.xfre.2024.08.009.

Validation of administrative health data for the identification of endometriosis diagnosis.

Kiser A, Hemmert R, Myrer R, Bucher B, Eilbeck K, Varner M Hum Reprod. 2024; 40(2):289-295.

PMID: 39704741 PMC: 11788219. DOI: 10.1093/humrep/deae281.

Examining the co-occurrence of endometriosis and polycystic ovarian syndrome.

Schliep K, Ghabayen L, Shaaban M, Hughes F, Pollack A, Stanford J AJOG Glob Rep. 2023; 3(3):100259.

PMID: 37663310 PMC: 10472311. DOI: 10.1016/j.xagr.2023.100259.

Estimation of multiple ordered ROC curves using placement values.

Ghosal S, Grantz K, Chen Z Stat Methods Med Res. 2022; 31(8):1470-1483.

PMID: 35450477 PMC: 9614716. DOI: 10.1177/09622802221094940.

Endometriosis classification, staging and reporting systems: a review on the road to a universally accepted endometriosis classification.

Vermeulen N, Abrao M, Einarsson J, Horne A, Johnson N, Lee T Hum Reprod Open. 2021; 2021(4):hoab025.

PMID: 34693032 PMC: 8530712. DOI: 10.1093/hropen/hoab025.

References

Buck Louis G, Hediger M, Peterson C, Croughan M, Sundaram R, Stanford J . Incidence of endometriosis by study population and diagnostic method: the ENDO study. Fertil Steril. 2011; 96(2):360-5. PMC: 3143230. DOI: 10.1016/j.fertnstert.2011.05.087. View

Buchweitz O, Wulfing P, Malik E . Interobserver variability in the diagnosis of minimal and mild endometriosis. Eur J Obstet Gynecol Reprod Biol. 2005; 122(2):213-7. DOI: 10.1016/j.ejogrb.2005.02.002. View

Adamson G . Endometriosis classification: an update. Curr Opin Obstet Gynecol. 2011; 23(4):213-20. DOI: 10.1097/GCO.0b013e328348a3ba. View

Ball E, Koh C, Janik G, Davis C . Gynaecological laparoscopy: 'see and treat' should be the gold standard. Curr Opin Obstet Gynecol. 2008; 20(4):325-30. DOI: 10.1097/GCO.0b013e32830002bb. View

Weijenborg P, Ter Kuile M, Jansen F . Intraobserver and interobserver reliability of videotaped laparoscopy evaluations for endometriosis and adhesions. Fertil Steril. 2006; 87(2):373-80. DOI: 10.1016/j.fertnstert.2006.06.052. View

. Revised American Society for Reproductive Medicine classification of endometriosis: 1996. Fertil Steril. 1997; 67(5):817-21. DOI: 10.1016/s0015-0282(97)81391-x. View

Coccia M, Rizzello F . Ultrasonographic staging: a new staging system for deep endometriosis. Ann N Y Acad Sci. 2011; 1221:61-9. DOI: 10.1111/j.1749-6632.2011.05951.x. View

Fraser I . Recognising, understanding and managing endometriosis. J Hum Reprod Sci. 2009; 1(2):56-64. PMC: 2700667. DOI: 10.4103/0974-1208.44112. View

Hornstein M, Gleason R, Orav J, Haas S, Friedman A, Rein M . The reproducibility of the revised American Fertility Society classification of endometriosis. Fertil Steril. 1993; 59(5):1015-21. View

10.

Rock J . The revised American Fertility Society classification of endometriosis: reproducibility of scoring. ZOLADEX Endometriosis Study Group. Fertil Steril. 1995; 63(5):1108-10. DOI: 10.1016/s0015-0282(16)57556-6. View

11.

Roberts C, Rock J . The current staging system for endometriosis: does it help?. Obstet Gynecol Clin North Am. 2003; 30(1):115-32. DOI: 10.1016/s0889-8545(02)00056-6. View

12.

. Endometriosis and infertility. Fertil Steril. 2006; 86(5 Suppl 1):S156-60. DOI: 10.1016/j.fertnstert.2006.08.014. View

13.

Landis J, Koch G . The measurement of observer agreement for categorical data. Biometrics. 1977; 33(1):159-74. View

14.

Kennedy S, Bergqvist A, Chapron C, DHooghe T, Dunselman G, Greb R . ESHRE guideline for the diagnosis and treatment of endometriosis. Hum Reprod. 2005; 20(10):2698-704. DOI: 10.1093/humrep/dei135. View

15.

. Classification of endometriosis. The American Fertility Society. Fertil Steril. 1979; 32(6):633-4. View

16.

Bland J, Altman D . Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986; 1(8476):307-10. View

17.

Adamson G, Pasta D . Endometriosis fertility index: the new, validated endometriosis staging system. Fertil Steril. 2009; 94(5):1609-15. DOI: 10.1016/j.fertnstert.2009.09.035. View

18.

Somigliana E, Vercellini P, Vigano P, Benaglia L, Crosignani P, Fedele L . Non-invasive diagnosis of endometriosis: the goal or own goal?. Hum Reprod. 2010; 25(8):1863-8. DOI: 10.1093/humrep/deq141. View

19.

. Revised American Fertility Society classification of endometriosis: 1985. Fertil Steril. 1985; 43(3):351-2. DOI: 10.1016/s0015-0282(16)48430-x. View

20.

Corson S, Batzer F, Gocial B, Kelly M, Gutmann J, Maislin G . Intra-observer and inter-observer variability in scoring laparoscopic diagnosis of pelvic adhesions. Hum Reprod. 1995; 10(1):161-4. DOI: 10.1093/humrep/10.1.161. View