PROC: an Open-source Package for R and S+ to Analyze and Compare ROC Curves

Overview

Journal BMC Bioinformatics

Publisher Biomed Central

Specialty Biology

Date 2011 Mar 19

PMID 21414208

Citations 5318

Authors

Xavier Robin

Natacha Turck

Alexandre Hainard

Natalia Tiberti

Frederique Lisacek

Jean-Charles Sanchez

Markus Muller

Affiliations

Soon will be listed here.

Abstract

Background: Receiver operating characteristic (ROC) curves are useful tools to evaluate classifiers in biomedical and bioinformatics applications. However, conclusions are often reached through inconsistent use or insufficient statistical analysis. To support researchers in their ROC curves analysis we developed pROC, a package for R and S+ that contains a set of tools displaying, analyzing, smoothing and comparing ROC curves in a user-friendly, object-oriented and flexible interface.

Results: With data previously imported into the R or S+ environment, the pROC package builds ROC curves and includes functions for computing confidence intervals, statistical tests for comparing total or partial area under the curve or the operating points of different classifiers, and methods for smoothing ROC curves. Intermediary and final results are visualised in user-friendly interfaces. A case study based on published clinical and biomarker data shows how to perform a typical ROC analysis with pROC.

Conclusions: pROC is a package for R and S+ specifically dedicated to ROC analysis. It proposes multiple statistical tests to compare ROC curves, and in particular partial areas under the curve, allowing proper ROC interpretation. pROC is available in two versions: in the R programming language or with a graphical user interface in the S+ statistical software. It is accessible at http://expasy.org/tools/pROC/ under the GNU General Public License. It is also distributed through the CRAN and CSAN public repositories, facilitating its installation.

Citing Articles

Combined transcriptomic and proteomic analyses reveal relevant myelin features in mice with ischemic stroke.

Qian Q, Lyu H, Wang W, Wang Q, Li D, Liu X Funct Integr Genomics. 2025; 25(1):64.

PMID: 40085348 DOI: 10.1007/s10142-025-01573-6.

A non-randomized pilot study to test the feasibility of developing a frailty scale for pet cats.

Colleran E, Delgado M, Ren Y, German A, Gruen M, Gunn-Moore D Front Vet Sci. 2025; 12:1549566.

PMID: 40078212 PMC: 11897749. DOI: 10.3389/fvets.2025.1549566.

Optimization and Benchmarking of RT-LAMP-CRISPR-Cas12a for the Detection of SARS-CoV-2 in Saliva.

Lynch C, Drummond R, Jelley L, Baker L, Smit E, Fleming R Int J Mol Sci. 2025; 26(5).

PMID: 40076433 PMC: 11899638. DOI: 10.3390/ijms26051806.

Classification of FAMACHA© Scores with Support Vector Machine Algorithm from Body Condition Score and Hematological Parameters in Pelibuey Sheep.

Torres-Chable O, Tirink C, Parra-Cortes R, Delgado M, Martinez I, Gomez-Vazquez A Animals (Basel). 2025; 15(5).

PMID: 40076020 PMC: 11898691. DOI: 10.3390/ani15050737.

Deep learning radiomics for the prediction of epidermal growth factor receptor mutation status based on MRI in brain metastasis from lung adenocarcinoma patients.

Cao P, Jia X, Wang X, Fan L, Chen Z, Zhao Y BMC Cancer. 2025; 25(1):443.

PMID: 40075375 PMC: 11899356. DOI: 10.1186/s12885-025-13823-8.

References

Zou K, Hall W, Shapiro D . Smooth non-parametric receiver operating characteristic (ROC) curves for continuous diagnostic tests. Stat Med. 1997; 16(19):2143-56. DOI: 10.1002/(sici)1097-0258(19971015)16:19<2143::aid-sim655>3.0.co;2-3. View

Robin X, Turck N, Hainard A, Lisacek F, Sanchez J, Muller M . Bioinformatics for protein biomarker panel classification: what is needed to bring biomarker panels into in vitro diagnostics?. Expert Rev Proteomics. 2009; 6(6):675-89. DOI: 10.1586/epr.09.83. View

Venkatraman E . A permutation test to compare receiver operating characteristic curves. Biometrics. 2000; 56(4):1134-8. DOI: 10.1111/j.0006-341x.2000.01134.x. View

Sonego P, Kocsor A, Pongor S . ROC analysis: applications to the classification of biological sequences and 3D structures. Brief Bioinform. 2008; 9(3):198-209. DOI: 10.1093/bib/bbm064. View

Sing T, Sander O, Beerenwinkel N, Lengauer T . ROCR: visualizing classifier performance in R. Bioinformatics. 2005; 21(20):3940-1. DOI: 10.1093/bioinformatics/bti623. View

Streiner D, Cairney J . What's under the ROC? An introduction to receiver operating characteristics curves. Can J Psychiatry. 2007; 52(2):121-8. DOI: 10.1177/070674370705200210. View

Braun T, Alonzo T . A modified sign test for comparing paired ROC curves. Biostatistics. 2007; 9(2):364-72. DOI: 10.1093/biostatistics/kxm036. View

Swets J . The Relative Operating Characteristic in Psychology: A technique for isolating effects of response bias finds wide use in the study of perception and cognition. Science. 1973; 182(4116):990-1000. DOI: 10.1126/science.182.4116.990. View

Hanley J . The robustness of the "binormal" assumptions used in fitting ROC curves. Med Decis Making. 1988; 8(3):197-203. DOI: 10.1177/0272989X8800800308. View

10.

DeLong E, Delong D, Clarke-Pearson D . Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988; 44(3):837-45. View

11.

Stephan C, Wesseling S, Schink T, Jung K . Comparison of eight computer programs for receiver-operating characteristic analysis. Clin Chem. 2003; 49(3):433-9. DOI: 10.1373/49.3.433. View

12.

Hanczar B, Hua J, Sima C, Weinstein J, Bittner M, Dougherty E . Small-sample precision of ROC-related estimates. Bioinformatics. 2010; 26(6):822-30. DOI: 10.1093/bioinformatics/btq037. View

13.

Jiang Y, Metz C, Nishikawa R . A receiver operating characteristic partial area index for highly sensitive diagnostic tests. Radiology. 1996; 201(3):745-50. DOI: 10.1148/radiology.201.3.8939225. View

14.

Hanley J, McNeil B . A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology. 1983; 148(3):839-43. DOI: 10.1148/radiology.148.3.6878708. View

15.

Turck N, Vutskits L, Sanchez-Pena P, Robin X, Hainard A, Gex-Fabry M . A multiparameter panel method for outcome prediction following aneurysmal subarachnoid hemorrhage. Intensive Care Med. 2009; 36(1):107-15. DOI: 10.1007/s00134-009-1641-y. View

16.

Bandos A, Rockette H, Gur D . A permutation test for comparing ROC curves in multireader studies a multi-reader ROC, permutation test. Acad Radiol. 2006; 13(4):414-20. DOI: 10.1016/j.acra.2005.12.012. View

17.

Bandos A, Rockette H, Gur D . A permutation test sensitive to differences in areas for comparing ROC curves from a paired design. Stat Med. 2005; 24(18):2873-93. DOI: 10.1002/sim.2149. View

18.

Pepe M, Longton G, Janes H . Estimation and Comparison of Receiver Operating Characteristic Curves. Stata J. 2010; 9(1):1. PMC: 2774909. View

19.

Metz C, Herman B, Shen J . Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data. Stat Med. 1998; 17(9):1033-53. DOI: 10.1002/(sici)1097-0258(19980515)17:9<1033::aid-sim784>3.0.co;2-z. View

20.

McClish D . Analyzing a portion of the ROC curve. Med Decis Making. 1989; 9(3):190-5. DOI: 10.1177/0272989X8900900307. View