A Prospective Evaluation of an Automated Classification System to Support Evidence-based Medicine and Systematic Review

Systematic reviews (SR) are an important and labor-intensive part of the Evidence-based Medicine process that could benefit from automated literature classification tools. We conducted a prospective study of a support vector machine-based classifier for supporting the SR literature triage process. Over 50,000 training data samples were collected for 18 topics prior to March 2008, and used to make predictions on 11,000 test data samples collected during the subsequent two years. Test performance (AUC) was comparable to that estimated by cross-validation on the training set, and ranging from 0.75 - 0.99. Mean AUC macro-averaged across all topics was 0.89, demonstrating that these methods can achieve accurate results in near-real world conditions and are promising tools for deployment to groups conducting SRs.