Systematic Benchmarking of Omics Computational Tools

Overview

Journal Nat Commun

Specialty Biology

Date 2019 Mar 29

PMID 30918265

Citations 61

Authors

Serghei Mangul

Lana S Martin

Brian L Hill

Angela Ka-Mei Lam

Margaret G Distler

Alex Zelikovsky

Eleazar Eskin

Jonathan Flint

Affiliations

Soon will be listed here.

Abstract

Computational omics methods packaged as software have become essential to modern biological research. The increasing dependence of scientists on these powerful software tools creates a need for systematic assessment of these methods, known as benchmarking. Adopting a standardized benchmarking practice could help researchers who use omics data to better leverage recent technological innovations. Our review summarizes benchmarking practices from 25 recent studies and discusses the challenges, advantages, and limitations of benchmarking across various domains of biology. We also propose principles that can make computational biology benchmarking studies more sustainable and reproducible, ultimately increasing the transparency of biomedical data and results.

Citing Articles

Genomic reproducibility in the bioinformatics era.

Baykal P, Labaj P, Markowetz F, Schriml L, Stekhoven D, Mangul S Genome Biol. 2024; 25(1):213.

PMID: 39123217 PMC: 11312195. DOI: 10.1186/s13059-024-03343-2.

Tackling neurodegeneration with omics: a path towards new targets and drugs.

Carraro C, Montgomery J, Klimmt J, Paquet D, Schultze J, Beyer M Front Mol Neurosci. 2024; 17:1414886.

PMID: 38952421 PMC: 11215216. DOI: 10.3389/fnmol.2024.1414886.

Packaging and containerization of computational methods.

Alser M, Lawlor B, Abdill R, Waymost S, Ayyala R, Rajkumar N Nat Protoc. 2024; 19(9):2529-2539.

PMID: 38565959 DOI: 10.1038/s41596-024-00986-0.

MAC-ErrorReads: machine learning-assisted classifier for filtering erroneous NGS reads.

Sami A, El-Metwally S, Rashad M BMC Bioinformatics. 2024; 25(1):61.

PMID: 38321434 PMC: 10848413. DOI: 10.1186/s12859-024-05681-1.

Challenges and best practices in omics benchmarking.

Brooks T, Lahens N, Mrcela A, Grant G Nat Rev Genet. 2024; 25(5):326-339.

PMID: 38216661 DOI: 10.1038/s41576-023-00679-6.

References

Huntley R, Sawford T, Martin M, ODonovan C . Understanding how and why the Gene Ontology and its annotations evolve: the GO within UniProt. Gigascience. 2014; 3(1):4. PMC: 3995153. DOI: 10.1186/2047-217X-3-4. View

Nookaew I, Papini M, Pornputtapong N, Scalcinati G, Fagerberg L, Uhlen M . A comprehensive comparison of RNA-Seq-based transcriptome analysis from reads to differential gene expression and cross-comparison with microarrays: a case study in Saccharomyces cerevisiae. Nucleic Acids Res. 2012; 40(20):10084-97. PMC: 3488244. DOI: 10.1093/nar/gks804. View

Clark W, Radivojac P . Information-theoretic evaluation of predicted ontological annotations. Bioinformatics. 2013; 29(13):i53-61. PMC: 3694662. DOI: 10.1093/bioinformatics/btt228. View

Thompson J, Linard B, Lecompte O, Poch O . A comprehensive benchmark study of multiple sequence alignment methods: current challenges and future perspectives. PLoS One. 2011; 6(3):e18093. PMC: 3069049. DOI: 10.1371/journal.pone.0018093. View

. A comprehensive assessment of RNA-seq accuracy, reproducibility and information content by the Sequencing Quality Control Consortium. Nat Biotechnol. 2014; 32(9):903-14. PMC: 4321899. DOI: 10.1038/nbt.2957. View

Li B, Dewey C . RSEM: accurate transcript quantification from RNA-Seq data with or without a reference genome. BMC Bioinformatics. 2011; 12:323. PMC: 3163565. DOI: 10.1186/1471-2105-12-323. View

Jiang Y, Ronnen Oron T, Clark W, Bankapur A, DAndrea D, Lepore R . An expanded evaluation of protein function prediction methods shows an improvement in accuracy. Genome Biol. 2016; 17(1):184. PMC: 5015320. DOI: 10.1186/s13059-016-1037-6. View

Artyomenko A, Wu N, Mangul S, Eskin E, Sun R, Zelikovsky A . Long Single-Molecule Reads Can Resolve the Complexity of the Influenza Virus Composed of Rare, Closely Related Mutant Variants. J Comput Biol. 2016; 24(6):558-570. PMC: 5467126. DOI: 10.1089/cmb.2016.0146. View

Tilstone C . DNA microarrays: vital statistics. Nature. 2003; 424(6949):610-2. DOI: 10.1038/424610a. View

10.

Saez-Rodriguez J, Costello J, Friend S, Kellen M, Mangravite L, Meyer P . Crowdsourcing biomedical research: leveraging communities as innovation engines. Nat Rev Genet. 2016; 17(8):470-86. PMC: 5918684. DOI: 10.1038/nrg.2016.69. View

11.

Langille M, Ravel J, Fricke W . "Available upon request": not good enough for microbiome data!. Microbiome. 2018; 6(1):8. PMC: 5761205. DOI: 10.1186/s40168-017-0394-z. View

12.

Smyrk T, Watson P, Kaul K, Lynch H . Tumor-infiltrating lymphocytes are a marker for microsatellite instability in colorectal carcinoma. Cancer. 2001; 91(12):2417-22. View

13.

Nagarajan N, Pop M . Sequence assembly demystified. Nat Rev Genet. 2013; 14(3):157-67. DOI: 10.1038/nrg3367. View

14.

Aniba M, Poch O, Thompson J . Issues in bioinformatics benchmarking: the case study of multiple sequence alignment. Nucleic Acids Res. 2010; 38(21):7353-63. PMC: 2995051. DOI: 10.1093/nar/gkq625. View

15.

Sczyrba A, Hofmann P, Belmann P, Koslicki D, Janssen S, Droge J . Critical Assessment of Metagenome Interpretation-a benchmark of metagenomics software. Nat Methods. 2017; 14(11):1063-1071. PMC: 5903868. DOI: 10.1038/nmeth.4458. View

16.

Nicolae M, Mangul S, Mandoiu I, Zelikovsky A . Estimation of alternative splicing isoform frequencies from RNA-Seq data. Algorithms Mol Biol. 2011; 6(1):9. PMC: 3107792. DOI: 10.1186/1748-7188-6-9. View

17.

Lindgreen S, Adair K, Gardner P . An evaluation of the accuracy and speed of metagenome analysis tools. Sci Rep. 2016; 6:19233. PMC: 4726098. DOI: 10.1038/srep19233. View

18.

Alberti C, Daniels N, Hernaez M, Voges J, Goldfeder R, Hernandez-Lopez A . An Evaluation Framework for Lossy Compression of Genome Sequencing Quality Values. Proc Data Compress Conf. 2017; 2016:221-230. PMC: 5568552. DOI: 10.1109/DCC.2016.39. View

19.

Altenhoff A, Boeckmann B, Capella-Gutierrez S, Dalquen D, DeLuca T, Forslund K . Standardized benchmarking in the quest for orthologs. Nat Methods. 2016; 13(5):425-30. PMC: 4827703. DOI: 10.1038/nmeth.3830. View

20.

Stodden V, Seiler J, Ma Z . An empirical analysis of journal policy effectiveness for computational reproducibility. Proc Natl Acad Sci U S A. 2018; 115(11):2584-2589. PMC: 5856507. DOI: 10.1073/pnas.1708290115. View