PipeComp, a General Framework for the Evaluation of Computational Pipelines, Reveals Performant Single Cell RNA-seq Preprocessing Tools

Overview

Journal Genome Biol

Specialties Biology
Genetics

Date 2020 Sep 3

PMID 32873325

Citations 46

Authors

Pierre-Luc Germain

Anthony Sonrel

Mark D Robinson

Affiliations

Soon will be listed here.

Abstract

We present pipeComp ( https://github.com/plger/pipeComp ), a flexible R framework for pipeline comparison handling interactions between analysis steps and relying on multi-level evaluation metrics. We apply it to the benchmark of single-cell RNA-sequencing analysis pipelines using simulated and real datasets with known cell identities, covering common methods of filtering, doublet detection, normalization, feature selection, denoising, dimensionality reduction, and clustering. pipeComp can easily integrate any other step, tool, or evaluation metric, allowing extensible benchmarks and easy applications to other fields, as we demonstrate through a study of the impact of removal of unwanted variation on differential expression analysis.

Citing Articles

Elucidating cardiomyocyte heterogeneity and maturation dynamics through integrated single-cell and spatial transcriptomics.

Wang X, Cao L, Chang R, Shen J, Ma L, Li Y iScience. 2025; 28(1):111596.

PMID: 39811652 PMC: 11732507. DOI: 10.1016/j.isci.2024.111596.

Unsupervised multi-scale clustering of single-cell transcriptomes to identify hierarchical structures of cell subtypes.

Song W, Ming C, Forst C, Zhang B Res Sq. 2025; .

PMID: 39764102 PMC: 11703337. DOI: 10.21203/rs.3.rs-5671748/v1.

High-dose intravenous BCG vaccination induces enhanced immune signaling in the airways.

Peters J, Irvine E, Makatsa M, Rosenberg J, Wadsworth 2nd M, Hughes T Sci Adv. 2025; 11(1):eadq8229.

PMID: 39742484 PMC: 11694782. DOI: 10.1126/sciadv.adq8229.

Statistically principled feature selection for single cell transcriptomics.

Dollinger E, Silkwood K, Atwood S, Nie Q, Lander A bioRxiv. 2024; .

PMID: 39463971 PMC: 11507810. DOI: 10.1101/2024.10.11.617709.

The estrogen response in fibroblasts promotes ovarian metastases of gastric cancer.

Hu S, Hu C, Xu J, Yu P, Yuan L, Li Z Nat Commun. 2024; 15(1):8447.

PMID: 39349474 PMC: 11443007. DOI: 10.1038/s41467-024-52615-9.

References

Salomon R, Kaczorowski D, Valdes-Mora F, Nordon R, Neild A, Farbehi N . Droplet-based single cell RNAseq tools: a practical guide. Lab Chip. 2019; 19(10):1706-1727. DOI: 10.1039/c8lc01239c. View

Townes F, Hicks S, Aryee M, Irizarry R . Feature selection and dimension reduction for single-cell RNA-Seq based on a multinomial model. Genome Biol. 2019; 20(1):295. PMC: 6927135. DOI: 10.1186/s13059-019-1861-6. View

Cole M, Risso D, Wagner A, DeTomaso D, Ngai J, Purdom E . Performance Assessment and Selection of Normalization Procedures for Single-Cell RNA-Seq. Cell Syst. 2019; 8(4):315-328.e8. PMC: 6544759. DOI: 10.1016/j.cels.2019.03.010. View

Zhang X, Li T, Liu F, Chen Y, Yao J, Li Z . Comparative Analysis of Droplet-Based Ultra-High-Throughput Single-Cell RNA-Seq Systems. Mol Cell. 2018; 73(1):130-142.e5. DOI: 10.1016/j.molcel.2018.10.020. View

Wang T, Li B, Nelson C, Nabavi S . Comparative analysis of differential gene expression analysis tools for single-cell RNA sequencing data. BMC Bioinformatics. 2019; 20(1):40. PMC: 6339299. DOI: 10.1186/s12859-019-2599-6. View

Tran H, Ang K, Chevrier M, Zhang X, Lee N, Goh M . A benchmark of batch-effect correction methods for single-cell RNA sequencing data. Genome Biol. 2020; 21(1):12. PMC: 6964114. DOI: 10.1186/s13059-019-1850-9. View

McGinnis C, Murrow L, Gartner Z . DoubletFinder: Doublet Detection in Single-Cell RNA Sequencing Data Using Artificial Nearest Neighbors. Cell Syst. 2019; 8(4):329-337.e4. PMC: 6853612. DOI: 10.1016/j.cels.2019.03.003. View

Leek J, Storey J . Capturing heterogeneity in gene expression studies by surrogate variable analysis. PLoS Genet. 2007; 3(9):1724-35. PMC: 1994707. DOI: 10.1371/journal.pgen.0030161. View

Satija R, Farrell J, Gennert D, Schier A, Regev A . Spatial reconstruction of single-cell gene expression data. Nat Biotechnol. 2015; 33(5):495-502. PMC: 4430369. DOI: 10.1038/nbt.3192. View

10.

Leek J, Johnson W, Parker H, Jaffe A, Storey J . The sva package for removing batch effects and other unwanted variation in high-throughput experiments. Bioinformatics. 2012; 28(6):882-3. PMC: 3307112. DOI: 10.1093/bioinformatics/bts034. View

11.

Svensson V, Beltrame E, Pachter L . A curated database reveals trends in single-cell transcriptomics. Database (Oxford). 2020; 2020. PMC: 7698659. DOI: 10.1093/database/baaa073. View

12.

Johnsson K, Soneson C, Fontes M . Low Bias Local Intrinsic Dimension Estimation from Expected Simplex Skewness. IEEE Trans Pattern Anal Mach Intell. 2015; 37(1):196-202. DOI: 10.1109/TPAMI.2014.2343220. View

13.

Germain P, Vitriolo A, Adamo A, Laise P, Das V, Testa G . RNAontheBENCH: computational and empirical resources for benchmarking RNAseq quantification and differential expression methods. Nucleic Acids Res. 2016; 44(11):5054-67. PMC: 4914128. DOI: 10.1093/nar/gkw448. View

14.

Love M, Huber W, Anders S . Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2. Genome Biol. 2014; 15(12):550. PMC: 4302049. DOI: 10.1186/s13059-014-0550-8. View

15.

Saelens W, Cannoodt R, Todorov H, Saeys Y . A comparison of single-cell trajectory inference methods. Nat Biotechnol. 2019; 37(5):547-554. DOI: 10.1038/s41587-019-0071-9. View

16.

Zappia L, Phipson B, Oshlack A . Exploring the single-cell RNA-seq analysis landscape with the scRNA-tools database. PLoS Comput Biol. 2018; 14(6):e1006245. PMC: 6034903. DOI: 10.1371/journal.pcbi.1006245. View

17.

Steinley D . Properties of the Hubert-Arabie adjusted Rand index. Psychol Methods. 2004; 9(3):386-96. DOI: 10.1037/1082-989X.9.3.386. View

18.

Hafemeister C, Satija R . Normalization and variance stabilization of single-cell RNA-seq data using regularized negative binomial regression. Genome Biol. 2019; 20(1):296. PMC: 6927181. DOI: 10.1186/s13059-019-1874-1. View

19.

. A comprehensive assessment of RNA-seq accuracy, reproducibility and information content by the Sequencing Quality Control Consortium. Nat Biotechnol. 2014; 32(9):903-14. PMC: 4321899. DOI: 10.1038/nbt.2957. View

20.

Bacher R, Chu L, Leng N, Gasch A, Thomson J, Stewart R . SCnorm: robust normalization of single-cell RNA-seq data. Nat Methods. 2017; 14(6):584-586. PMC: 5473255. DOI: 10.1038/nmeth.4263. View