Inter-expert and Intra-expert Agreement on the Diagnosis and Treatment of Retinopathy of Prematurity

Purpose: To evaluate inter-expert and intra-expert agreement on the diagnosis and treatment of retinopathy of prematurity (ROP).

Design: Prospective intra- and inter-rater reliability analysis.

Methods: In this multicenter study, 260 wide-field digital photographs of 52 patients were presented to 7 recognized ROP experts on 2 consecutive assessment days 8 weeks apart. Experts were asked to assess the patients for ROP stage, presence of plus disease, presence of aggressive posterior ROP, necessity for treatment, and suggested treatment. Agreement levels were measured with Fleiss' kappa and Cohen's kappa.

Results: Inter-expert agreement was fair for the ROP stage (κ = 0.24), plus disease (κ = 0.32), and aggressive posterior ROP (κ = 0.35); moderate for the necessity for treatment (κ = 0.41); and fair for the kind of treatment (κ = 0.38). Perfect inter-expert agreement was found in 9.6% of all patients for ROP stage 0-5, 45.1% for ≥ stage 2 ROP, 17.3% for plus disease, 57.7% for aggressive posterior ROP, and 25% for the necessity for treatment. Intra-expert agreement was higher than inter-expert agreement and was moderate for the ROP stage (κ = 0.56) and plus disease (κ = 0.51), moderate to substantial for aggressive posterior ROP (κ = 0.60), moderate for the necessity for treatment (κ = 0.47), and substantial for the kind of treatment (κ = 0.63).

Conclusions: ROP diagnosis and treatment decisions differ between experts and by 1 expert made on different days, indicating that the grading process is subjective and there is an observer bias when diagnosing ROP. These results could influence current practice in ROP assessment and training, and prompt further refinement of international ROP guidelines.