Benchmarking Computational Variant Effect Predictors by Their Ability to Infer Human Traits

Overview

Journal Genome Biol

Specialties Biology
Genetics

Date 2024 Jul 2

PMID 38951922

Authors

Daniel R Tabet

Da Kuang

Megan C Lancaster

Roujia Li

Karen Liu

Jochen Weile

Atina G Cote

Yingzhou Wu

Robert A Hegele

Dan M Roden

Frederick P Roth

Affiliations

Soon will be listed here.

Abstract

Background: Computational variant effect predictors offer a scalable and increasingly reliable means of interpreting human genetic variation, but concerns of circularity and bias have limited previous methods for evaluating and comparing predictors. Population-level cohorts of genotyped and phenotyped participants that have not been used in predictor training can facilitate an unbiased benchmarking of available methods. Using a curated set of human gene-trait associations with a reported rare-variant burden association, we evaluate the correlations of 24 computational variant effect predictors with associated human traits in the UK Biobank and All of Us cohorts.

Results: AlphaMissense outperformed all other predictors in inferring human traits based on rare missense variants in UK Biobank and All of Us participants. The overall rankings of computational variant effect predictors in these two cohorts showed a significant positive correlation.

Conclusion: We describe a method to assess computational variant effect predictors that sidesteps the limitations of previous evaluations. This approach is generalizable to future predictors and could continue to inform predictor choice for personal and clinical genetics.

References

Vaser R, Adusumalli S, Leng S, Sikic M, Ng P . SIFT missense predictions for genomes. Nat Protoc. 2015; 11(1):1-9. DOI: 10.1038/nprot.2015.123. View

Cirulli E, White S, Read R, Elhanan G, Metcalf W, Tanudjaja F . Genome-wide rare variant analysis for thousands of phenotypes in over 70,000 exomes from two cohorts. Nat Commun. 2020; 11(1):542. PMC: 6987107. DOI: 10.1038/s41467-020-14288-y. View

Chun S, Fay J . Identification of deleterious mutations within three human genomes. Genome Res. 2009; 19(9):1553-61. PMC: 2752137. DOI: 10.1101/gr.092619.109. View

Karczewski K, Francioli L, Tiao G, Cummings B, Alfoldi J, Wang Q . The mutational constraint spectrum quantified from variation in 141,456 humans. Nature. 2020; 581(7809):434-443. PMC: 7334197. DOI: 10.1038/s41586-020-2308-7. View

Richards S, Aziz N, Bale S, Bick D, Das S, Gastier-Foster J . Standards and guidelines for the interpretation of sequence variants: a joint consensus recommendation of the American College of Medical Genetics and Genomics and the Association for Molecular Pathology. Genet Med. 2015; 17(5):405-24. PMC: 4544753. DOI: 10.1038/gim.2015.30. View

Dong C, Wei P, Jian X, Gibbs R, Boerwinkle E, Wang K . Comparison and integration of deleteriousness prediction methods for nonsynonymous SNVs in whole exome sequencing studies. Hum Mol Genet. 2015; 24(8):2125-37. PMC: 4375422. DOI: 10.1093/hmg/ddu733. View

Povysil G, Petrovski S, Hostyk J, Aggarwal V, Allen A, Goldstein D . Rare-variant collapsing analyses for complex traits: guidelines and applications. Nat Rev Genet. 2019; 20(12):747-759. DOI: 10.1038/s41576-019-0177-4. View

Cirulli E, Goldstein D . Uncovering the roles of rare variants in common disease through whole-genome sequencing. Nat Rev Genet. 2010; 11(6):415-25. DOI: 10.1038/nrg2779. View

Schubach M, Maass T, Nazaretyan L, Roner S, Kircher M . CADD v1.7: using protein language models, regulatory CNNs and other nucleotide-level scores to improve genome-wide variant predictions. Nucleic Acids Res. 2024; 52(D1):D1143-D1154. PMC: 10767851. DOI: 10.1093/nar/gkad989. View

10.

Ionita-Laza I, McCallum K, Xu B, Buxbaum J . A spectral approach integrating functional genomic annotations for coding and noncoding variants. Nat Genet. 2016; 48(2):214-20. PMC: 4731313. DOI: 10.1038/ng.3477. View

11.

Grimm D, Azencott C, Aicheler F, Gieraths U, MacArthur D, Samocha K . The evaluation of tools used to predict the impact of missense variants is hindered by two types of circularity. Hum Mutat. 2015; 36(5):513-23. PMC: 4409520. DOI: 10.1002/humu.22768. View

12.

Hobbs H, Brown M, Goldstein J . Molecular genetics of the LDL receptor gene in familial hypercholesterolemia. Hum Mutat. 1992; 1(6):445-66. DOI: 10.1002/humu.1380010602. View

13.

Shihab H, Gough J, Cooper D, Stenson P, Barker G, Edwards K . Predicting the functional, molecular, and phenotypic consequences of amino acid substitutions using hidden Markov models. Hum Mutat. 2012; 34(1):57-65. PMC: 3558800. DOI: 10.1002/humu.22225. View

14.

Katsonis P, Wilhelm K, Williams A, Lichtarge O . Genome interpretation using in silico predictors of variant impact. Hum Genet. 2022; 141(10):1549-1577. PMC: 9055222. DOI: 10.1007/s00439-022-02457-6. View

15.

Pejaver V, Urresti J, Lugo-Martinez J, Pagel K, Lin G, Nam H . Inferring the molecular and phenotypic impact of amino acid variants with MutPred2. Nat Commun. 2020; 11(1):5918. PMC: 7680112. DOI: 10.1038/s41467-020-19669-x. View

16.

Pujar S, OLeary N, Farrell C, Loveland J, Mudge J, Wallin C . Consensus coding sequence (CCDS) database: a standardized set of human and mouse protein-coding regions supported by expert curation. Nucleic Acids Res. 2017; 46(D1):D221-D228. PMC: 5753299. DOI: 10.1093/nar/gkx1031. View

17.

Kuang D, Weile J, Kishore N, Nguyen M, Rubin A, Fields S . MaveRegistry: a collaboration platform for multiplexed assays of variant effect. Bioinformatics. 2021; 37(19):3382-3383. PMC: 8504617. DOI: 10.1093/bioinformatics/btab215. View

18.

Liu X, Li C, Mou C, Dong Y, Tu Y . dbNSFP v4: a comprehensive database of transcript-specific functional predictions and annotations for human nonsynonymous and splice-site SNVs. Genome Med. 2020; 12(1):103. PMC: 7709417. DOI: 10.1186/s13073-020-00803-9. View

19.

Livesey B, Marsh J . Using deep mutational scanning to benchmark variant effect predictors and identify disease mutations. Mol Syst Biol. 2020; 16(7):e9380. PMC: 7336272. DOI: 10.15252/msb.20199380. View

20.

Riesselman A, Ingraham J, Marks D . Deep generative models of genetic variation capture the effects of mutations. Nat Methods. 2018; 15(10):816-822. PMC: 6693876. DOI: 10.1038/s41592-018-0138-4. View