Inter-rater Reliability for Witnessed Collapse and Presence of Bystander CPR

Objective: Witnessed collapse and bystander CPR are the variables most frequently associated with good outcome from out-of-hospital cardiac arrest (OOHCA). The reliability of abstracting witnessed collapse and bystander CPR from prehospital Emergency Medical Services (EMS) patient care records (PCRs) is not known. We sought to determine the inter-rater reliability for different methods of ascertaining and defining witnessed collapse and performance of bystander CPR.

Methods: A sample of 100 PCRs for patients with OOHCA was selected at random from a pool of 325 PCRs between May 2003 and January 2005. Paramedics used a drop down menu to indicate witnessed collapse and bystander CPR, and completed a narrative description of the event. An on-scene EMS physician also completed a data sheet. The PCR was examined by two separate evaluators to determine the presence of witnessed collapse and bystander CPR. A consensus was reached by three other reviewers using all available data sources. Inter-rater agreement was quantified using the unweighted kappa statistic.

Results: For witnessed collapse, there is substantial agreement between the following: individual evaluators (kappa=0.76, S.D.=0.07), individual evaluators and consensus group (kappa=0.61, S.D.=0.07 and 0.66, S.D.=0.07), and physician and consensus group (kappa=0.68, S.D.=0.08). Agreement between individual evaluators and the physician was fair to moderate (kappa=0.38, S.D.=0.07 and 0.44, S.D.=0.07). Agreement between individual evaluators, physician, consensus group and the PCR drop down menu was fair to moderate (kappa range 0.33, S.D.=0.09 to 0.54, S.D.=0.09). For bystander CPR, there is substantial agreement between the individual evaluators and the consensus group (kappa=0.64, S.D.=0.07 and 0.63, S.D.=0.06) and between the physician and the consensus group (kappa=0.61, S.D.=0.08). Agreement between the two individual evaluators is moderate (kappa=0.59, S.D.=0.07). Agreement between the physician and individual evaluators is fair (kappa=0.36, S.D.=0.07 and 0.38, S.D.=0.07). The PCR drop down menu had moderate to substantial agreement with the individual evaluators, physician, and consensus group (kappa range 0.50, S.D.=0.09 to 0.75, S.D.=0.09).

Conclusions: Determination of witnessed collapse and bystander CPR during OOHCA may be less reliable than previously thought, and differences between methods of rating could influence study results.