Assessing the Role of Evolutionary Information for Enhancing Protein Language Model Embeddings

Overview

Journal Sci Rep

Specialty Science

Date 2024 Sep 5

PMID 39237735

Authors

Kyra Erckert

Burkhard Rost

Affiliations

Soon will be listed here.

Abstract

Embeddings from protein Language Models (pLMs) are replacing evolutionary information from multiple sequence alignments (MSAs) as the most successful input for protein prediction. Is this because embeddings capture evolutionary information? We tested various approaches to explicitly incorporate evolutionary information into embeddings on various protein prediction tasks. While older pLMs (SeqVec, ProtBert) significantly improved through MSAs, the more recent pLM ProtT5 did not benefit. For most tasks, pLM-based outperformed MSA-based methods, and the combination of both even decreased performance for some (intrinsic disorder). We highlight the effectiveness of pLM-based methods and find limited benefits from integrating MSAs.

References

Radivojac P, Obradovic Z, Brown C, Dunker A . Improving sequence alignments for intrinsically disordered proteins. Pac Symp Biocomput. 2002; :589-600. View

Huang H, Sarai A . Analysis of the relationships between evolvability, thermodynamics, and the functions of intrinsically disordered proteins/regions. Comput Biol Chem. 2012; 41:51-7. DOI: 10.1016/j.compbiolchem.2012.10.001. View

Heinzinger M, Elnaggar A, Wang Y, Dallago C, Nechaev D, Matthes F . Modeling aspects of the language of life through transfer-learning protein sequences. BMC Bioinformatics. 2019; 20(1):723. PMC: 6918593. DOI: 10.1186/s12859-019-3220-8. View

Littmann M, Heinzinger M, Dallago C, Weissenow K, Rost B . Protein embeddings and deep learning predict binding residues for various ligand classes. Sci Rep. 2021; 11(1):23916. PMC: 8668950. DOI: 10.1038/s41598-021-03431-4. View

Lange J, Wyrwicz L, Vriend G . KMAD: knowledge-based multiple sequence alignment for intrinsically disordered proteins. Bioinformatics. 2015; 32(6):932-6. PMC: 4803389. DOI: 10.1093/bioinformatics/btv663. View

Suzek B, Wang Y, Huang H, McGarvey P, Wu C . UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searches. Bioinformatics. 2014; 31(6):926-32. PMC: 4375400. DOI: 10.1093/bioinformatics/btu739. View

Rost B, Sander C . Prediction of protein secondary structure at better than 70% accuracy. J Mol Biol. 1993; 232(2):584-99. DOI: 10.1006/jmbi.1993.1413. View

Littmann M, Heinzinger M, Dallago C, Olenyi T, Rost B . Embeddings from deep learning transfer GO annotations beyond homology. Sci Rep. 2021; 11(1):1160. PMC: 7806674. DOI: 10.1038/s41598-020-80786-0. View

Dunker A, Babu M, Barbar E, Blackledge M, Bondos S, Dosztanyi Z . What's in a name? Why these proteins are intrinsically disordered: Why these proteins are intrinsically disordered. Intrinsically Disord Proteins. 2017; 1(1):e24157. PMC: 5424803. DOI: 10.4161/idp.24157. View

10.

Schlessinger A, Rost B . Protein flexibility and rigidity predicted from sequence. Proteins. 2005; 61(1):115-26. DOI: 10.1002/prot.20587. View

11.

Michel M, Hayat S, Skwark M, Sander C, Marks D, Elofsson A . PconsFold: improved contact predictions improve protein models. Bioinformatics. 2014; 30(17):i482-8. PMC: 4147911. DOI: 10.1093/bioinformatics/btu458. View

12.

Nallapareddy V, Bordin N, Sillitoe I, Heinzinger M, Littmann M, Waman V . CATHe: detection of remote homologues for CATH superfamilies using embeddings from protein language models. Bioinformatics. 2023; 39(1). PMC: 9887088. DOI: 10.1093/bioinformatics/btad029. View

13.

Lin Z, Akin H, Rao R, Hie B, Zhu Z, Lu W . Evolutionary-scale prediction of atomic-level protein structure with a language model. Science. 2023; 379(6637):1123-1130. DOI: 10.1126/science.ade2574. View

14.

Newport T, Sansom M, Stansfeld P . The MemProtMD database: a resource for membrane-embedded protein structures and their lipid interactions. Nucleic Acids Res. 2018; 47(D1):D390-D397. PMC: 6324062. DOI: 10.1093/nar/gky1047. View

15.

Laine E, Karami Y, Carbone A . GEMME: A Simple and Fast Global Epistatic Model Predicting Mutational Effects. Mol Biol Evol. 2019; 36(11):2604-2619. PMC: 6805226. DOI: 10.1093/molbev/msz179. View

16.

Jones D, Singh T, Kosciolek T, Tetchner S . MetaPSICOV: combining coevolution methods for accurate prediction of contacts and long range hydrogen bonding in proteins. Bioinformatics. 2014; 31(7):999-1006. PMC: 4382908. DOI: 10.1093/bioinformatics/btu791. View

17.

Liu J, Rost B . NORSp: Predictions of long regions without regular secondary structure. Nucleic Acids Res. 2003; 31(13):3833-5. PMC: 168922. DOI: 10.1093/nar/gkg515. View

18.

Wells J, Hawkins-Hooker A, Bordin N, Sillitoe I, Paige B, Orengo C . Chainsaw: protein domain segmentation with fully convolutional neural networks. Bioinformatics. 2024; 40(5). PMC: 11256964. DOI: 10.1093/bioinformatics/btae296. View

19.

Hayat S, Peters C, Shu N, Tsirigos K, Elofsson A . Inclusion of dyad-repeat pattern improves topology prediction of transmembrane β-barrel proteins. Bioinformatics. 2016; 32(10):1571-3. DOI: 10.1093/bioinformatics/btw025. View

20.

Steinegger M, Soding J . MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nat Biotechnol. 2017; 35(11):1026-1028. DOI: 10.1038/nbt.3988. View