The Evaluation of Tools Used to Predict the Impact of Missense Variants is Hindered by Two Types of Circularity

Overview

Journal Hum Mutat

Specialty Genetics

Date 2015 Feb 17

PMID 25684150

Citations 182

Authors

Dominik G Grimm

Chloe-Agathe Azencott

Fabian Aicheler

Udo Gieraths

Daniel G MacArthur

Kaitlin E Samocha

David N Cooper

Peter D Stenson

Mark J Daly

Jordan W Smoller

Laramie E Duncan

Karsten M Borgwardt

Affiliations

Soon will be listed here.

Abstract

Prioritizing missense variants for further experimental investigation is a key challenge in current sequencing studies for exploring complex and Mendelian diseases. A large number of in silico tools have been employed for the task of pathogenicity prediction, including PolyPhen-2, SIFT, FatHMM, MutationTaster-2, MutationAssessor, Combined Annotation Dependent Depletion, LRT, phyloP, and GERP++, as well as optimized methods of combining tool scores, such as Condel and Logit. Due to the wealth of these methods, an important practical question to answer is which of these tools generalize best, that is, correctly predict the pathogenic character of new variants. We here demonstrate in a study of 10 tools on five datasets that such a comparative evaluation of these tools is hindered by two types of circularity: they arise due to (1) the same variants or (2) different variants from the same protein occurring both in the datasets used for training and for evaluation of these tools, which may lead to overly optimistic results. We show that comparative evaluations of predictors that do not address these types of circularity may erroneously conclude that circularity confounded tools are most accurate among all tools, and may even outperform optimized combinations of tools.

Citing Articles

The specification game: rethinking the evaluation of drug response prediction for precision oncology.

Codice F, Pancotti C, Rollo C, Moreau Y, Fariselli P, Raimondi D J Cheminform. 2025; 17(1):33.

PMID: 40087708 DOI: 10.1186/s13321-025-00972-y.

AFFIPred: AlphaFold2 structure-based Functional Impact Prediction of missense variations.

Pir M, Timucin E Protein Sci. 2025; 34(2):e70030.

PMID: 39840793 PMC: 11751861. DOI: 10.1002/pro.70030.

Meta-EA: a gene-specific combination of available computational tools for predicting missense variant effects.

Katsonis P, Lichtarge O Nat Commun. 2025; 16(1):159.

PMID: 39746940 PMC: 11696468. DOI: 10.1038/s41467-024-55066-4.

Estimating the proportion of beneficial mutations that are not adaptive in mammals.

Latrille T, Joseph J, Hartasanchez D, Salamin N PLoS Genet. 2024; 20(12):e1011536.

PMID: 39724093 PMC: 11709321. DOI: 10.1371/journal.pgen.1011536.

Development and validation of animal variant classification guidelines to objectively evaluate genetic variant pathogenicity in domestic animals.

Boeykens F, Abitbol M, Anderson H, Casselman I, de Citres C, Hayward J Front Vet Sci. 2024; 11:1497817.

PMID: 39703406 PMC: 11656590. DOI: 10.3389/fvets.2024.1497817.

References

Gough J, Karplus K, Hughey R, Chothia C . Assignment of homology to genome sequences using a library of hidden Markov models that represent all proteins of known structure. J Mol Biol. 2001; 313(4):903-19. DOI: 10.1006/jmbi.2001.5080. View

Ng P, Henikoff S . SIFT: Predicting amino acid changes that affect protein function. Nucleic Acids Res. 2003; 31(13):3812-4. PMC: 168916. DOI: 10.1093/nar/gkg509. View

Sonnhammer E, Eddy S, Durbin R . Pfam: a comprehensive database of protein domain families based on seed alignments. Proteins. 1997; 28(3):405-20. DOI: 10.1002/(sici)1097-0134(199707)28:3<405::aid-prot10>3.0.co;2-l. View

Hubbard T, Aken B, Ayling S, Ballester B, Beal K, Bragin E . Ensembl 2009. Nucleic Acids Res. 2008; 37(Database issue):D690-7. PMC: 2686571. DOI: 10.1093/nar/gkn828. View

Hindorff L, Sethupathy P, Junkins H, Ramos E, Mehta J, Collins F . Potential etiologic and functional implications of genome-wide association loci for human diseases and traits. Proc Natl Acad Sci U S A. 2009; 106(23):9362-7. PMC: 2687147. DOI: 10.1073/pnas.0903103106. View

Chun S, Fay J . Identification of deleterious mutations within three human genomes. Genome Res. 2009; 19(9):1553-61. PMC: 2752137. DOI: 10.1101/gr.092619.109. View

Camacho C, Coulouris G, Avagyan V, Ma N, Papadopoulos J, Bealer K . BLAST+: architecture and applications. BMC Bioinformatics. 2009; 10:421. PMC: 2803857. DOI: 10.1186/1471-2105-10-421. View

Mottaz A, David F, Veuthey A, Yip Y . Easy retrieval of single amino-acid polymorphisms and phenotype information using SwissVar. Bioinformatics. 2010; 26(6):851-2. PMC: 2832822. DOI: 10.1093/bioinformatics/btq028. View

Adzhubei I, Schmidt S, Peshkin L, Ramensky V, Gerasimova A, Bork P . A method and server for predicting damaging missense mutations. Nat Methods. 2010; 7(4):248-9. PMC: 2855889. DOI: 10.1038/nmeth0410-248. View

10.

McLaren W, Pritchard B, Rios D, Chen Y, Flicek P, Cunningham F . Deriving the consequences of genomic variants with the Ensembl API and SNP Effect Predictor. Bioinformatics. 2010; 26(16):2069-70. PMC: 2916720. DOI: 10.1093/bioinformatics/btq330. View

11.

Wang K, Li M, Hakonarson H . ANNOVAR: functional annotation of genetic variants from high-throughput sequencing data. Nucleic Acids Res. 2010; 38(16):e164. PMC: 2938201. DOI: 10.1093/nar/gkq603. View

12.

Davydov E, Goode D, Sirota M, Cooper G, Sidow A, Batzoglou S . Identifying a high fraction of the human genome to be under selective constraint using GERP++. PLoS Comput Biol. 2010; 6(12):e1001025. PMC: 2996323. DOI: 10.1371/journal.pcbi.1001025. View

13.

Thusberg J, Olatubosun A, Vihinen M . Performance of mutation pathogenicity prediction methods on missense variants. Hum Mutat. 2011; 32(4):358-68. DOI: 10.1002/humu.21445. View

14.

Magrane M . UniProt Knowledgebase: a hub of integrated protein data. Database (Oxford). 2011; 2011:bar009. PMC: 3070428. DOI: 10.1093/database/bar009. View

15.

Gonzalez-Perez A, Lopez-Bigas N . Improving the assessment of the outcome of nonsynonymous SNVs with a consensus deleteriousness score, Condel. Am J Hum Genet. 2011; 88(4):440-9. PMC: 3071923. DOI: 10.1016/j.ajhg.2011.03.004. View

16.

Cooper G, Shendure J . Needles in stacks of needles: finding disease-causal variants in a wealth of genomic data. Nat Rev Genet. 2011; 12(9):628-40. DOI: 10.1038/nrg3046. View

17.

Reva B, Antipin Y, Sander C . Predicting the functional impact of protein mutations: application to cancer genomics. Nucleic Acids Res. 2011; 39(17):e118. PMC: 3177186. DOI: 10.1093/nar/gkr407. View

18.

Stenson P, Mort M, Ball E, Shaw K, Phillips A, Cooper D . The Human Gene Mutation Database: building a comprehensive mutation repository for clinical and molecular genetics, diagnostic testing and personalized genomic medicine. Hum Genet. 2013; 133(1):1-9. PMC: 3898141. DOI: 10.1007/s00439-013-1358-4. View

19.

Purcell S, Moran J, Fromer M, Ruderfer D, Solovieff N, Roussos P . A polygenic burden of rare disruptive mutations in schizophrenia. Nature. 2014; 506(7487):185-90. PMC: 4136494. DOI: 10.1038/nature12975. View

20.

Leongamornlert D, Mahmud N, Tymrakiewicz M, Saunders E, Dadaev T, Castro E . Germline BRCA1 mutations increase prostate cancer risk. Br J Cancer. 2012; 106(10):1697-701. PMC: 3349179. DOI: 10.1038/bjc.2012.146. View