The Receiver Operating Characteristic Curve Accurately Assesses Imbalanced Datasets

Overview

Journal Patterns (N Y)

Date 2024 Jul 15

PMID 39005487

Authors

Eve Richardson

Raphael Trevizani

Jason A Greenbaum

Hannah Carter

Morten Nielsen

Bjoern Peters

Affiliations

Soon will be listed here.

Abstract

Many problems in biology require looking for a "needle in a haystack," corresponding to a binary classification where there are a few positives within a much larger set of negatives, which is referred to as a class imbalance. The receiver operating characteristic (ROC) curve and the associated area under the curve (AUC) have been reported as ill-suited to evaluate prediction performance on imbalanced problems where there is more interest in performance on the positive minority class, while the precision-recall (PR) curve is preferable. We show via simulation and a real case study that this is a misinterpretation of the difference between the ROC and PR spaces, showing that the ROC curve is robust to class imbalance, while the PR curve is highly sensitive to class imbalance. Furthermore, we show that class imbalance cannot be easily disentangled from classifier performance measured via PR-AUC.

Citing Articles

Using a Neural Network Architecture for the Prediction of Neurologic Outcome for Out-of-Hospital Cardiac Arrests Using Hospital Level Variables and Novel Physiologic Markers.

Razo M, Kotini P, Li J, Khosla S, Buhimschi I, Hoek T Bioengineering (Basel). 2025; 12(2).

PMID: 40001644 PMC: 11852285. DOI: 10.3390/bioengineering12020124.

Comparative Analysis of Recurrent Neural Networks with Conjoint Fingerprints for Skin Corrosion Prediction.

Duy H, Srisongkram T J Chem Inf Model. 2025; 65(3):1305-1317.

PMID: 39835935 PMC: 11815816. DOI: 10.1021/acs.jcim.4c02062.

Application of machine learning for detecting high fall risk in middle-aged workers using video-based analysis of the first 3 steps.

Sakane N, Yamauchi K, Kutsuna I, Suganuma A, Domichi M, Hirano K J Occup Health. 2025; 67(1).

PMID: 39792357 PMC: 11848130. DOI: 10.1093/joccuh/uiae075.

Assessing Glioblastoma Treatment Response Using Machine Learning Approach Based on Magnetic Resonance Images Radiomics: An Exploratory Study.

Sadeghinasab A, Fatahiasl J, Tahmasbi M, Razmjoo S, Yousefipour M Health Sci Rep. 2025; 8(1):e70323.

PMID: 39741746 PMC: 11683675. DOI: 10.1002/hsr2.70323.

Understanding the heterogeneous performance of variant effect predictors across human protein-coding genes.

Fawzy M, Marsh J Sci Rep. 2024; 14(1):26114.

PMID: 39478110 PMC: 11526010. DOI: 10.1038/s41598-024-76202-6.

References

Williams N, Rodrigues C, Truong J, Ascher D, Holien J . DockNet: high-throughput protein-protein interface contact prediction. Bioinformatics. 2022; 39(1). PMC: 9825772. DOI: 10.1093/bioinformatics/btac797. View

Pittala S, Bailey-Kellogg C . Learning context-aware structural representations to predict antigen and antibody binding interfaces. Bioinformatics. 2020; 36(13):3996-4003. PMC: 7332568. DOI: 10.1093/bioinformatics/btaa263. View

Li Y, Sackett P, Nielsen M, Barra C . NetAllergen, a random forest model integrating MHC-II presentation propensity for improved allergenicity prediction. Bioinform Adv. 2023; 3(1):vbad151. PMC: 10603389. DOI: 10.1093/bioadv/vbad151. View

Swamidass S, Azencott C, Daily K, Baldi P . A CROC stronger than ROC: measuring, visualizing and optimizing early retrieval. Bioinformatics. 2010; 26(10):1348-56. PMC: 2865862. DOI: 10.1093/bioinformatics/btq140. View

Chicco D, Jurman G . The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics. 2020; 21(1):6. PMC: 6941312. DOI: 10.1186/s12864-019-6413-7. View

Chinery L, Wahome N, Moal I, Deane C . Paragraph-antibody paratope prediction using graph neural networks with minimal feature vectors. Bioinformatics. 2022; 39(1). DOI: 10.1093/bioinformatics/btac732. View

Jeni L, Cohn J, de la Torre F . Facing Imbalanced Data Recommendations for the Use of Performance Metrics. Int Conf Affect Comput Intell Interact Workshops. 2015; 2013:245-251. PMC: 4285355. DOI: 10.1109/ACII.2013.47. View

Gainza P, Sverrisson F, Monti F, Rodola E, Boscaini D, Bronstein M . Deciphering interaction fingerprints from protein molecular surfaces using geometric deep learning. Nat Methods. 2019; 17(2):184-192. DOI: 10.1038/s41592-019-0666-6. View

Clifford J, Hoie M, Deleuran S, Peters B, Nielsen M, Marcatili P . BepiPred-3.0: Improved B-cell epitope prediction using protein language models. Protein Sci. 2022; 31(12):e4497. PMC: 9679979. DOI: 10.1002/pro.4497. View

10.

Clark R, Webster-Clark D . Managing bias in ROC curves. J Comput Aided Mol Des. 2008; 22(3-4):141-6. DOI: 10.1007/s10822-008-9181-z. View

11.

Saito T, Rehmsmeier M . The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS One. 2015; 10(3):e0118432. PMC: 4349800. DOI: 10.1371/journal.pone.0118432. View

12.

Nilsson J, Kaabinejadian S, Yari H, Peters B, Barra C, Gragert L . Machine learning reveals limited contribution of trans-only encoded variants to the HLA-DQ immunopeptidome. Commun Biol. 2023; 6(1):442. PMC: 10121683. DOI: 10.1038/s42003-023-04749-7. View

13.

Carter H, Chen S, Isik L, Tyekucheva S, Velculescu V, Kinzler K . Cancer-specific high-throughput annotation of somatic mutations: computational prediction of driver missense mutations. Cancer Res. 2009; 69(16):6660-7. PMC: 2763410. DOI: 10.1158/0008-5472.CAN-09-1133. View

14.

Ozenne B, Subtil F, Maucort-Boulch D . The precision--recall curve overcame the optimism of the receiver operating characteristic curve in rare diseases. J Clin Epidemiol. 2015; 68(8):855-9. DOI: 10.1016/j.jclinepi.2015.02.010. View

15.

Daberdaku S, Ferrari C . Antibody interface prediction with 3D Zernike descriptors and SVM. Bioinformatics. 2018; 35(11):1870-1876. DOI: 10.1093/bioinformatics/bty918. View

16.

Dai B, Bailey-Kellogg C . Protein interaction interface region prediction by geometric deep learning. Bioinformatics. 2021; 37(17):2580-2588. PMC: 8428585. DOI: 10.1093/bioinformatics/btab154. View

17.

Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez J . pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics. 2011; 12:77. PMC: 3068975. DOI: 10.1186/1471-2105-12-77. View

18.

McClish D . Analyzing a portion of the ROC curve. Med Decis Making. 1989; 9(3):190-5. DOI: 10.1177/0272989X8900900307. View

19.

Swets J . Measuring the accuracy of diagnostic systems. Science. 1988; 240(4857):1285-93. DOI: 10.1126/science.3287615. View

20.

Stranzl T, Larsen M, Lundegaard C, Nielsen M . NetCTLpan: pan-specific MHC class I pathway epitope predictions. Immunogenetics. 2010; 62(6):357-68. PMC: 2875469. DOI: 10.1007/s00251-010-0441-4. View