Gene Set Enrichment for Reproducible Science: Comparison of CERNO and Eight Other Algorithms

Overview

Journal Bioinformatics

Publisher Oxford University Press

Specialty Biology

Date 2019 Jun 6

PMID 31165139

Citations 58

Authors

Joanna Zyla

Michal Marczyk

Teresa Domaszewska

Stefan H E Kaufmann

Joanna Polanska

January Weiner

Affiliations

Soon will be listed here.

Abstract

Motivation: Analysis of gene set (GS) enrichment is an essential part of functional omics studies. Here, we complement the established evaluation metrics of GS enrichment algorithms with a novel approach to assess the practical reproducibility of scientific results obtained from GS enrichment tests when applied to related data from different studies.

Results: We evaluated eight established and one novel algorithm for reproducibility, sensitivity, prioritization, false positive rate and computational time. In addition to eight established algorithms, we also included Coincident Extreme Ranks in Numerical Observations (CERNO), a flexible and fast algorithm based on modified Fisher P-value integration. Using real-world datasets, we demonstrate that CERNO is robust to ranking metrics, as well as sample and GS size. CERNO had the highest reproducibility while remaining sensitive, specific and fast. In the overall ranking Pathway Analysis with Down-weighting of Overlapping Genes, CERNO and over-representation analysis performed best, while CERNO and GeneSetTest scored high in terms of reproducibility.

Availability And Implementation: tmod package implementing the CERNO algorithm is available from CRAN (cran.r-project.org/web/packages/tmod/index.html) and an online implementation can be found at http://tmod.online/. The datasets analyzed in this study are widely available in the KEGGdzPathwaysGEO, KEGGandMetacoreDzPathwaysGEO R package and GEO repository.

Supplementary Information: Supplementary data are available at Bioinformatics online.

Citing Articles

Gene Set Enrichment Analysis in Zebrafish Embryos Is Susceptible to False-Positive Results in the Absence of Differentially Expressed Genes.

Stead J, Lee H, Williams A, Ramirez S, Atlas E, Mennigen J Bioinform Biol Insights. 2025; 19:11779322251321071.

PMID: 40040651 PMC: 11877468. DOI: 10.1177/11779322251321071.

Divergent endothelial mechanisms drive arteriovenous malformations in Alk1 and SMAD4 loss-of-function.

Oppenheim O, Giese W, Park H, Baumann E, Ivanov A, Beule D bioRxiv. 2025; .

PMID: 39829872 PMC: 11741317. DOI: 10.1101/2025.01.03.631070.

Integrative multiomics reveals common endotypes across PSEN1, PSEN2, and APP mutations in familial Alzheimer's disease.

Valdes P, Caldwell A, Liu Q, Fitzgerald M, Ramachandran S, Karch C Alzheimers Res Ther. 2025; 17(1):5.

PMID: 39754192 PMC: 11699654. DOI: 10.1186/s13195-024-01659-6.

Molecular profiles, sources and lineage restrictions of stem cells in an annelid regeneration model.

Stockinger A, Adelmann L, Fahrenberger M, Ruta C, Ozpolat B, Milivojev N Nat Commun. 2024; 15(1):9882.

PMID: 39557833 PMC: 11574210. DOI: 10.1038/s41467-024-54041-3.

Insights into phylogenetic relationships and gene rearrangements: complete mitogenomes of two sympatric species in the genus (Anura, Ranidae).

Li J, Xie M, Zhang F, Shu J, Zhang J, Zhang Z Zookeys. 2024; 1216:63-82.

PMID: 39474245 PMC: 11519660. DOI: 10.3897/zookeys.1216.131847.

References

Weiner 3rd J, Maertzdorf J, Sutherland J, Duffy F, Thompson E, Suliman S . Metabolite changes in blood predict the onset of tuberculosis. Nat Commun. 2018; 9(1):5208. PMC: 6283869. DOI: 10.1038/s41467-018-07635-7. View

Tarca A, Draghici S, Khatri P, Hassan S, Mittal P, Kim J . A novel signaling pathway impact analysis. Bioinformatics. 2008; 25(1):75-82. PMC: 2732297. DOI: 10.1093/bioinformatics/btn577. View

Edgar R, Domrachev M, Lash A . Gene Expression Omnibus: NCBI gene expression and hybridization array data repository. Nucleic Acids Res. 2001; 30(1):207-10. PMC: 99122. DOI: 10.1093/nar/30.1.207. View

Tamayo P, Steinhardt G, Liberzon A, Mesirov J . The limitations of simple gene set enrichment analysis assuming gene independence. Stat Methods Med Res. 2012; 25(1):472-87. PMC: 3758419. DOI: 10.1177/0962280212460441. View

Powers R, Goodspeed A, Pielke-Lombardo H, Tan A, Costello J . GSEA-InContext: identifying novel and common patterns in expression experiments. Bioinformatics. 2018; 34(13):i555-i564. PMC: 6022535. DOI: 10.1093/bioinformatics/bty271. View

Zyla J, Marczyk M, Weiner J, Polanska J . Ranking metrics in gene set enrichment analysis: do they matter?. BMC Bioinformatics. 2017; 18(1):256. PMC: 5427619. DOI: 10.1186/s12859-017-1674-0. View

Ihnatova I, Popovici V, Budinska E . A critical comparison of topology-based pathway analysis methods. PLoS One. 2018; 13(1):e0191154. PMC: 5784953. DOI: 10.1371/journal.pone.0191154. View

Ritchie J, Pantazatos S, French L . Transcriptomic characterization of MRI contrast with focus on the T1-w/T2-w ratio in the cerebral cortex. Neuroimage. 2018; 174:504-517. PMC: 6450807. DOI: 10.1016/j.neuroimage.2018.03.027. View

Santoro F, Pettini E, Kazmin D, Ciabattini A, Fiorino F, Gilfillan G . Transcriptomics of the Vaccine Immune Response: Priming With Adjuvant Modulates Recall Innate Responses After Boosting. Front Immunol. 2018; 9:1248. PMC: 5996052. DOI: 10.3389/fimmu.2018.01248. View

10.

Subramanian A, Tamayo P, Mootha V, Mukherjee S, Ebert B, Gillette M . Gene set enrichment analysis: a knowledge-based approach for interpreting genome-wide expression profiles. Proc Natl Acad Sci U S A. 2005; 102(43):15545-50. PMC: 1239896. DOI: 10.1073/pnas.0506580102. View

11.

Tarca A, Draghici S, Bhatti G, Romero R . Down-weighting overlapping genes improves gene set analysis. BMC Bioinformatics. 2012; 13:136. PMC: 3443069. DOI: 10.1186/1471-2105-13-136. View

12.

Kunnath-Velayudhan S, Salamon H, Wang H, Davidow A, Molina D, Huynh V . Dynamic antibody responses to the Mycobacterium tuberculosis proteome. Proc Natl Acad Sci U S A. 2010; 107(33):14703-8. PMC: 2930474. DOI: 10.1073/pnas.1009080107. View

13.

Jaakkola M, McGlinchey A, Klen R, Elo L . PASI: A novel pathway method to identify delicate group effects. PLoS One. 2018; 13(7):e0199991. PMC: 6033442. DOI: 10.1371/journal.pone.0199991. View

14.

Gu Z, Liu J, Cao K, Zhang J, Wang J . Centrality-based pathway enrichment: a systematic approach for finding significant pathways dominated by key genes. BMC Syst Biol. 2012; 6:56. PMC: 3443660. DOI: 10.1186/1752-0509-6-56. View

15.

Esterhuyse M, Weiner 3rd J, Caron E, Loxton A, Iannaccone M, Wagman C . Epigenetics and Proteomics Join Transcriptomics in the Quest for Tuberculosis Biomarkers. mBio. 2015; 6(5):e01187-15. PMC: 4600108. DOI: 10.1128/mBio.01187-15. View

16.

Shojaie A, Michailidis G . Network enrichment analysis in complex experiments. Stat Appl Genet Mol Biol. 2010; 9:Article22. PMC: 2898649. DOI: 10.2202/1544-6115.1483. View

17.

Tarca A, Bhatti G, Romero R . A comparison of gene set analysis methods in terms of sensitivity, prioritization and specificity. PLoS One. 2013; 8(11):e79217. PMC: 3829842. DOI: 10.1371/journal.pone.0079217. View

18.

Jaakkola M, Elo L . Empirical comparison of structure-based pathway methods. Brief Bioinform. 2015; 17(2):336-45. PMC: 4793894. DOI: 10.1093/bib/bbv049. View

19.

Godec J, Tan Y, Liberzon A, Tamayo P, Bhattacharya S, Butte A . Compendium of Immune Signatures Identifies Conserved and Species-Specific Biology in Response to Inflammation. Immunity. 2016; 44(1):194-206. PMC: 5330663. DOI: 10.1016/j.immuni.2015.12.006. View

20.

Foroutan M, Bhuva D, Lyu R, Horan K, Cursons J, Davis M . Single sample scoring of molecular phenotypes. BMC Bioinformatics. 2018; 19(1):404. PMC: 6219008. DOI: 10.1186/s12859-018-2435-4. View