Inter-rater Reliability of a Computerized Presenting-complaint-linked Triage System in an Urban Emergency Department

Background: Triage reliability studies typically use hypothetical scenarios and weighted kappa scores where agreement within one level is considered satisfactory. But if triage category is used to help define ED case-mix groups for comparative or benchmarking processes, agreement on exact triage level and major system involved is important. Our hypothesis was that a computerized menu that links presenting complaints to preferred triage levels (PC-linked triage) would provide high triage reliability.

Objectives: Our objective was to assess inter-rater reliability of PC-linked triage using the Canadian Emergency Department Triage and Acuity Scale (CTAS) in a real-time clinical setting, considering agreement on exact triage level and primary body system involved.

Methods: On duty triage nurses entered patient presenting complaint and PC-linked triage level as per standard procedure. In a convenience sample of patients, a second nurse, blinded to triage assignment, observed the triage interaction and independently entered presenting complaint and triage level on a dummy terminal.

Results: During the study, 15 nurse pairs triaged 266 patients. Study patients matched actual emergency department case mix closely. Triage nurses agreed exactly in 74% of cases and within one level in 94% of cases. The unweighted kappa value was 0.66 (95% confidence interval [CI], 0.60-0.73) and the quadratic weighted kappa value was 0.75 (95% CI, 0.68-0.81). Kappa for agreement on major system involved was 0.80 (95% CI, 0.69-0.91).

Conclusion: PC-linked triage has high inter-rater reliability in a real-time clinical setting. PC-linked triage may be useful as one factor in defining case-mix groups for benchmarking and comparative purposes.