Uncertainty-aware Single-cell Annotation with a Hierarchical Reject Option

Overview

Journal Bioinformatics

Publisher Oxford University Press

Specialty Biology

Date 2024 Mar 5

PMID 38441258

Authors

Lauren Theunissen

Thomas Mortier

Yvan Saeys

Willem Waegeman

Affiliations

Soon will be listed here.

Abstract

Motivation: Automatic cell type annotation methods assign cell type labels to new datasets by extracting relationships from a reference RNA-seq dataset. However, due to the limited resolution of gene expression features, there is always uncertainty present in the label assignment. To enhance the reliability and robustness of annotation, most machine learning methods address this uncertainty by providing a full reject option, i.e. when the predicted confidence score of a cell type label falls below a user-defined threshold, no label is assigned and no prediction is made. As a better alternative, some methods deploy hierarchical models and consider a so-called partial rejection by returning internal nodes of the hierarchy as label assignment. However, because a detailed experimental analysis of various rejection approaches is missing in the literature, there is currently no consensus on best practices.

Results: We evaluate three annotation approaches (i) full rejection, (ii) partial rejection, and (iii) no rejection for both flat and hierarchical probabilistic classifiers. Our findings indicate that hierarchical classifiers are superior when rejection is applied, with partial rejection being the preferred rejection approach, as it preserves a significant amount of label information. For optimal rejection implementation, the rejection threshold should be determined through careful examination of a method's rejection behavior. Without rejection, flat and hierarchical annotation perform equally well, as long as the cell type hierarchy accurately captures transcriptomic relationships.

Availability And Implementation: Code is freely available at https://github.com/Latheuni/Hierarchical_reject and https://doi.org/10.5281/zenodo.10697468.

References

Li H, Janssens J, De Waegeneer M, Kolluru S, Davie K, Gardeux V . Fly Cell Atlas: A single-nucleus transcriptomic atlas of the adult fruit fly. Science. 2022; 375(6584):eabk2432. PMC: 8944923. DOI: 10.1126/science.abk2432. View

Costa M, Reeve S, Grumbling G, Osumi-Sutherland D . The Drosophila anatomy ontology. J Biomed Semantics. 2013; 4(1):32. PMC: 4015547. DOI: 10.1186/2041-1480-4-32. View

Wang S, Pisco A, McGeever A, Brbic M, Zitnik M, Darmanis S . Leveraging the Cell Ontology to classify unseen cell types. Nat Commun. 2021; 12(1):5556. PMC: 8455606. DOI: 10.1038/s41467-021-25725-x. View

Kaymaz Y, Ganglberger F, Tang M, Haslinger C, Fernandez-Albert F, Lawless N . HieRFIT: a hierarchical cell type classification tool for projections from complex single-cell atlas datasets. Bioinformatics. 2021; 37(23):4431-4436. DOI: 10.1093/bioinformatics/btab499. View

Cao Z, Wei L, Lu S, Yang D, Gao G . Searching large-scale scRNA-seq databases via unbiased cell embedding with Cell BLAST. Nat Commun. 2020; 11(1):3458. PMC: 7351785. DOI: 10.1038/s41467-020-17281-7. View

Mortier T, Wieme A, Vandamme P, Waegeman W . Bacterial species identification using MALDI-TOF mass spectrometry and machine learning techniques: A large-scale benchmarking study. Comput Struct Biotechnol J. 2021; 19:6157-6168. PMC: 8649224. DOI: 10.1016/j.csbj.2021.11.004. View

Kiselev V, Yiu A, Hemberg M . scmap: projection of single-cell RNA-seq data across data sets. Nat Methods. 2018; 15(5):359-362. DOI: 10.1038/nmeth.4644. View

Michielsen L, Reinders M, Mahfouz A . Hierarchical progressive learning of cell identities in single-cell data. Nat Commun. 2021; 12(1):2799. PMC: 8121839. DOI: 10.1038/s41467-021-23196-8. View

Abdelaal T, Michielsen L, Cats D, Hoogduin D, Mei H, Reinders M . A comparison of automatic cell identification methods for single-cell RNA sequencing data. Genome Biol. 2019; 20(1):194. PMC: 6734286. DOI: 10.1186/s13059-019-1795-z. View

10.

Heumos L, Schaar A, Lance C, Litinetskaya A, Drost F, Zappia L . Best practices for single-cell analysis across modalities. Nat Rev Genet. 2023; 24(8):550-572. PMC: 10066026. DOI: 10.1038/s41576-023-00586-w. View

11.

Bernstein M, Ma Z, Gleicher M, Dewey C . CellO: comprehensive and hierarchical cell type classification of human cells with the Cell Ontology. iScience. 2020; 24(1):101913. PMC: 7753962. DOI: 10.1016/j.isci.2020.101913. View

12.

Stuart T, Butler A, Hoffman P, Hafemeister C, Papalexi E, Mauck 3rd W . Comprehensive Integration of Single-Cell Data. Cell. 2019; 177(7):1888-1902.e21. PMC: 6687398. DOI: 10.1016/j.cell.2019.05.031. View

13.

Pliner H, Shendure J, Trapnell C . Supervised classification enables rapid annotation of cell atlases. Nat Methods. 2019; 16(10):983-986. PMC: 6791524. DOI: 10.1038/s41592-019-0535-3. View

14.

Galdos F, Xu S, Goodyer W, Duan L, Huang Y, Lee S . devCellPy is a machine learning-enabled pipeline for automated annotation of complex multilayered single-cell transcriptomic data. Nat Commun. 2022; 13(1):5271. PMC: 9452519. DOI: 10.1038/s41467-022-33045-x. View

15.

Satija R, Farrell J, Gennert D, Schier A, Regev A . Spatial reconstruction of single-cell gene expression data. Nat Biotechnol. 2015; 33(5):495-502. PMC: 4430369. DOI: 10.1038/nbt.3192. View

16.

Lin Y, Cao Y, Kim H, Salim A, Speed T, Lin D . scClassify: sample size estimation and multiscale classification of cells using single and multiple reference. Mol Syst Biol. 2020; 16(6):e9389. PMC: 7306901. DOI: 10.15252/msb.20199389. View

17.

Wu Z, Wu H . Accounting for cell type hierarchy in evaluating single cell RNA-seq clustering. Genome Biol. 2020; 21(1):123. PMC: 7249323. DOI: 10.1186/s13059-020-02027-x. View

18.

Elmentaite R, Conde C, Yang L, Teichmann S . Single-cell atlases: shared and tissue-specific cell types across human organs. Nat Rev Genet. 2022; 23(7):395-410. DOI: 10.1038/s41576-022-00449-w. View

19.

Alquicira-Hernandez J, Sathe A, Ji H, Nguyen Q, Powell J . scPred: accurate supervised method for cell-type classification from single-cell RNA-seq data. Genome Biol. 2019; 20(1):264. PMC: 6907144. DOI: 10.1186/s13059-019-1862-5. View

20.

Gramates L, Agapite J, Attrill H, Calvi B, Crosby M, Dos Santos G . FlyBase: a guided tour of highlighted features. Genetics. 2022; 220(4). PMC: 8982030. DOI: 10.1093/genetics/iyac035. View