Critical Assessment of Metagenome Interpretation-a Benchmark of Metagenomics Software

Methods for assembly, taxonomic profiling and binning are key to interpreting metagenome data, but a lack of consensus about benchmarking complicates performance assessment. The Critical Assessment of Metagenome Interpretation (CAMI) challenge has engaged the global developer community to benchmark their programs on highly complex and realistic data sets, generated from ∼700 newly sequenced microorganisms and ∼600 novel viruses and plasmids and representing common experimental setups. Assembly and genome binning programs performed well for species represented by individual genomes but were substantially affected by the presence of related strains. Taxonomic profiling and binning programs were proficient at high taxonomic ranks, with a notable performance decrease below family level. Parameter settings markedly affected performance, underscoring their importance for program reproducibility. The CAMI results highlight current challenges but also provide a roadmap for software selection to answer specific research questions.

Citing Articles

Semisynthetic simulation for microbiome data analysis.

Sankaran K, Kodikara S, Li J, Le Cao K Brief Bioinform. 2025; 26(1).

PMID: 39927858 PMC: 11808806. DOI: 10.1093/bib/bbaf051.

Community assembly modeling of microbial evolution within Barrett's esophagus and esophageal adenocarcinoma.

Guccione C, Sfiligoi I, Gonzalez A, Shaffer J, Kazachkova M, Weng Y bioRxiv. 2025; .

PMID: 39868296 PMC: 11760701. DOI: 10.1101/2025.01.14.633020.

Taming large-scale genomic analyses via sparsified genomics.

Alser M, Eudine J, Mutlu O Nat Commun. 2025; 16(1):876.

PMID: 39837860 PMC: 11751491. DOI: 10.1038/s41467-024-55762-1.

Integrative genomics would strengthen AMR understanding through ONE health approach.

Liu C, Pandey R Heliyon. 2025; 10(14):e34719.

PMID: 39816336 PMC: 11734142. DOI: 10.1016/j.heliyon.2024.e34719.

A sensitivity analysis of methodological variables associated with microbiome measurements.

Forry S, Servetas S, Dootz J, Hunter M, Kralj J, Filliben J Microbiol Spectr. 2025; 13(2):e0069624.

PMID: 39807898 PMC: 11792480. DOI: 10.1128/spectrum.00696-24.

References

Coil D, Jospin G, Darling A . A5-miseq: an updated pipeline to assemble microbial genomes from Illumina MiSeq data. Bioinformatics. 2014; 31(4):587-9. DOI: 10.1093/bioinformatics/btu661. View

Wu Y, Simmons B, Singer S . MaxBin 2.0: an automated binning algorithm to recover genomes from multiple metagenomic datasets. Bioinformatics. 2015; 32(4):605-7. DOI: 10.1093/bioinformatics/btv638. View

Bendall M, Stevens S, Chan L, Malfatti S, Schwientek P, Tremblay J . Genome-wide selective sweeps and gene-specific sweeps in natural bacterial populations. ISME J. 2016; 10(7):1589-601. PMC: 4918448. DOI: 10.1038/ismej.2015.241. View

Segata N, Waldron L, Ballarini A, Narasimhan V, Jousson O, Huttenhower C . Metagenomic microbial community profiling using unique clade-specific marker genes. Nat Methods. 2012; 9(8):811-4. PMC: 3443552. DOI: 10.1038/nmeth.2066. View

Kashtan N, Roggensack S, Rodrigue S, Thompson J, Biller S, Coe A . Single-cell genomics reveals hundreds of coexisting subpopulations in wild Prochlorococcus. Science. 2014; 344(6182):416-20. DOI: 10.1126/science.1248575. View

Wilke A, Bischof J, Gerlach W, Glass E, Harrison T, Keegan K . The MG-RAST metagenomics database and portal in 2015. Nucleic Acids Res. 2015; 44(D1):D590-4. PMC: 4702923. DOI: 10.1093/nar/gkv1322. View

Arumugam M, Raes J, Pelletier E, Le Paslier D, Yamada T, Mende D . Enterotypes of the human gut microbiome. Nature. 2011; 473(7346):174-80. PMC: 3728647. DOI: 10.1038/nature09944. View

Li S, Zhu A, Benes V, Costea P, Hercog R, Hildebrand F . Durable coexistence of donor and recipient strains after fecal microbiota transplantation. Science. 2016; 352(6285):586-9. DOI: 10.1126/science.aad8852. View

Koren S, Phillippy A . One chromosome, one contig: complete microbial genomes from long-read sequencing and assembly. Curr Opin Microbiol. 2014; 23:110-20. DOI: 10.1016/j.mib.2014.11.014. View

10.

Lin H, Liao Y . Accurate binning of metagenomic contigs via automated clustering sequences using information of genomic signatures and marker genes. Sci Rep. 2016; 6:24175. PMC: 4828714. DOI: 10.1038/srep24175. View

11.

Chen I, Markowitz V, Chu K, Palaniappan K, Szeto E, Pillay M . IMG/M: integrated genome and metagenome comparative data analysis system. Nucleic Acids Res. 2016; 45(D1):D507-D516. PMC: 5210632. DOI: 10.1093/nar/gkw929. View

12.

Balvociute M, Huson D . SILVA, RDP, Greengenes, NCBI and OTT - how do these taxonomies compare?. BMC Genomics. 2017; 18(Suppl 2):114. PMC: 5374703. DOI: 10.1186/s12864-017-3501-4. View

13.

. Structure, function and diversity of the healthy human microbiome. Nature. 2012; 486(7402):207-14. PMC: 3564958. DOI: 10.1038/nature11234. View

14.

Nguyen N, Mirarab S, Liu B, Pop M, Warnow T . TIPP: taxonomic identification and phylogenetic profiling. Bioinformatics. 2014; 30(24):3548-55. PMC: 4253836. DOI: 10.1093/bioinformatics/btu721. View

15.

Silva G, Cuevas D, Dutilh B, Edwards R . FOCUS: an alignment-free model to identify organisms in metagenomes using non-negative least squares. PeerJ. 2014; 2:e425. PMC: 4060023. DOI: 10.7717/peerj.425. View

16.

Koslicki D, Falush D . MetaPalette: a -mer Painting Approach for Metagenomic Taxonomic Profiling and Quantification of Novel Strain Variation. mSystems. 2016; 1(3). PMC: 5069763. DOI: 10.1128/mSystems.00020-16. View

17.

Mikheenko A, Saveliev V, Gurevich A . MetaQUAST: evaluation of metagenome assemblies. Bioinformatics. 2015; 32(7):1088-90. DOI: 10.1093/bioinformatics/btv697. View

18.

Strous M, Kraft B, Bisdorf R, Tegetmeyer H . The binning of metagenomic contigs for microbial physiology of mixed cultures. Front Microbiol. 2012; 3:410. PMC: 3514610. DOI: 10.3389/fmicb.2012.00410. View

19.

Kang D, Froula J, Egan R, Wang Z . MetaBAT, an efficient tool for accurately reconstructing single genomes from complex microbial communities. PeerJ. 2015; 3:e1165. PMC: 4556158. DOI: 10.7717/peerj.1165. View

20.

Koslicki D, Foucart S, Rosen G . Quikr: a method for rapid reconstruction of bacterial communities via compressive sensing. Bioinformatics. 2013; 29(17):2096-102. DOI: 10.1093/bioinformatics/btt336. View