CodonBERT Large Language Model for MRNA Vaccines

Overview

Journal Genome Res

Specialty Genetics

Date 2024 Jul 1

PMID 38951026

Authors

Sizhen Li

Saeed Moayedpour

Ruijiang Li

Michael Bailey

Saleh Riahi

Lorenzo Kogler-Anele

Milad Miladi

Jacob Miner

Fabien Pertuy

Dinghai Zheng

Jun Wang

Akshay Balsubramani

Khang Tran

Minnie Zacharia

Monica Wu

Xiaobo Gu

Ryan Clinton

Carla Asquith

Joseph Skaleski

Lianne Boeglin

Sudha Chivukula

Anusha Dias

Tod Strugnell

Fernando Ulloa Montoya

Vikram Agarwal

Ziv Bar-Joseph

Sven Jager

Affiliations

Soon will be listed here.

Abstract

mRNA-based vaccines and therapeutics are gaining popularity and usage across a wide range of conditions. One of the critical issues when designing such mRNAs is sequence optimization. Even small proteins or peptides can be encoded by an enormously large number of mRNAs. The actual mRNA sequence can have a large impact on several properties, including expression, stability, immunogenicity, and more. To enable the selection of an optimal sequence, we developed CodonBERT, a large language model (LLM) for mRNAs. Unlike prior models, CodonBERT uses codons as inputs, which enables it to learn better representations. CodonBERT was trained using more than 10 million mRNA sequences from a diverse set of organisms. The resulting model captures important biological concepts. CodonBERT can also be extended to perform prediction tasks for various mRNA properties. CodonBERT outperforms previous mRNA prediction methods, including on a new flu vaccine data set.

Citing Articles

PlasGO: enhancing GO-based function prediction for plasmid-encoded proteins based on genetic structure.

Ji Y, Shang J, Guan J, Zou W, Liao H, Tang X Gigascience. 2024; 13.

PMID: 39704702 PMC: 11659980. DOI: 10.1093/gigascience/giae104.

Advanced technologies for the development of infectious disease vaccines.

Gupta A, Rudra A, Reed K, Langer R, Anderson D Nat Rev Drug Discov. 2024; 23(12):914-938.

PMID: 39433939 DOI: 10.1038/s41573-024-01041-z.

A Suite of Foundation Models Captures the Contextual Interplay Between Codons.

Naghipourfar M, Chen S, Howard M, Macdonald C, Saberi A, Hagen T bioRxiv. 2024; .

PMID: 39416097 PMC: 11482952. DOI: 10.1101/2024.10.10.617568.

Are genomic language models all you need? Exploring genomic language models on protein downstream tasks.

Boshar S, Trop E, de Almeida B, Copoiu L, Pierrot T Bioinformatics. 2024; 40(9).

PMID: 39212609 PMC: 11399231. DOI: 10.1093/bioinformatics/btae529.

Predicting the translation efficiency of messenger RNA in mammalian cells.

Zheng D, Persyn L, Wang J, Liu Y, Montoya F, Cenik C bioRxiv. 2024; .

PMID: 39149337 PMC: 11326250. DOI: 10.1101/2024.08.11.607362.

References

Agarwal V, Shendure J . Predicting mRNA Abundance Directly from Genomic Sequence Using Deep Convolutional Neural Networks. Cell Rep. 2020; 31(7):107663. DOI: 10.1016/j.celrep.2020.107663. View

Jackson L, Anderson E, Rouphael N, Roberts P, Makhene M, Coler R . An mRNA Vaccine against SARS-CoV-2 - Preliminary Report. N Engl J Med. 2020; 383(20):1920-1931. PMC: 7377258. DOI: 10.1056/NEJMoa2022483. View

Wayment-Steele H, Kladwang W, Watkins A, Kim D, Tunguz B, Reade W . Deep learning models for predicting RNA degradation via dual crowdsourcing. Nat Mach Intell. 2022; 4(12):1174-1184. PMC: 9771809. DOI: 10.1038/s42256-022-00571-8. View

Aw J, Shen Y, Wilm A, Sun M, Lim X, Boon K . In Vivo Mapping of Eukaryotic RNA Interactomes Reveals Principles of Higher-Order Organization and Regulation. Mol Cell. 2016; 62(4):603-17. DOI: 10.1016/j.molcel.2016.04.028. View

Leppek K, Byeon G, Kladwang W, Wayment-Steele H, Kerr C, Xu A . Combinatorial optimization of mRNA structure, stability, and translation for RNA-based therapeutics. Nat Commun. 2022; 13(1):1536. PMC: 8940940. DOI: 10.1038/s41467-022-28776-w. View

Mauger D, Cabral B, Presnyak V, Su S, Reid D, Goodman B . mRNA structure regulates protein expression through changes in functional half-life. Proc Natl Acad Sci U S A. 2019; 116(48):24075-24083. PMC: 6883848. DOI: 10.1073/pnas.1908052116. View

Ji Y, Zhou Z, Liu H, Davuluri R . DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome. Bioinformatics. 2021; 37(15):2112-2120. PMC: 11025658. DOI: 10.1093/bioinformatics/btab083. View

Lorentzen C, Haanen J, Met O, Marie Svane I . Clinical advances and ongoing trials on mRNA vaccines for cancer treatment. Lancet Oncol. 2022; 23(10):e450-e458. PMC: 9512276. DOI: 10.1016/S1470-2045(22)00372-2. View

Rives A, Meier J, Sercu T, Goyal S, Lin Z, Liu J . Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proc Natl Acad Sci U S A. 2021; 118(15). PMC: 8053943. DOI: 10.1073/pnas.2016239118. View

10.

Li S, Zhang H, Zhang L, Liu K, Liu B, Mathews D . LinearTurboFold: Linear-time global prediction of conserved structures for RNA homologs with applications to SARS-CoV-2. Proc Natl Acad Sci U S A. 2021; 118(52). PMC: 8719904. DOI: 10.1073/pnas.2116269118. View

11.

Groher F, Bofill-Bosch C, Schneider C, Braun J, Jager S, Geissler K . Riboswitching with ciprofloxacin-development and characterization of a novel RNA regulator. Nucleic Acids Res. 2018; 46(4):2121-2132. PMC: 5829644. DOI: 10.1093/nar/gkx1319. View

12.

Wheeler D, Barrett T, Benson D, Bryant S, Canese K, Chetvernin V . Database resources of the National Center for Biotechnology Information. Nucleic Acids Res. 2006; 35(Database issue):D5-12. PMC: 1781113. DOI: 10.1093/nar/gkl1031. View

13.

Mauro V, Chappell S . A critical analysis of codon optimization in human therapeutics. Trends Mol Med. 2014; 20(11):604-13. PMC: 4253638. DOI: 10.1016/j.molmed.2014.09.003. View

14.

Nieuwkoop T, Terlouw B, Stevens K, Scheltema R, de Ridder D, van der Oost J . Revealing determinants of translation efficiency via whole-gene codon randomization and machine learning. Nucleic Acids Res. 2023; 51(5):2363-2376. PMC: 10018363. DOI: 10.1093/nar/gkad035. View

15.

Maruggi G, Chiarot E, Giovani C, Buccato S, Bonacci S, Frigimelica E . Immunogenicity and protective efficacy induced by self-amplifying mRNA vaccines encoding bacterial antigens. Vaccine. 2016; 35(2):361-368. DOI: 10.1016/j.vaccine.2016.11.040. View

16.

Schlake T, Thess A, Fotin-Mleczek M, Kallen K . Developing mRNA-vaccine technologies. RNA Biol. 2012; 9(11):1319-30. PMC: 3597572. DOI: 10.4161/rna.22269. View

17.

Mauro V . Codon Optimization in the Production of Recombinant Biotherapeutics: Potential Risks and Considerations. BioDrugs. 2018; 32(1):69-81. DOI: 10.1007/s40259-018-0261-x. View

18.

Groher A, Jager S, Schneider C, Groher F, Hamacher K, Suess B . Tuning the Performance of Synthetic Riboswitches using Machine Learning. ACS Synth Biol. 2018; 8(1):34-44. DOI: 10.1021/acssynbio.8b00207. View

19.

Grigoriev I, Nikitin R, Haridas S, Kuo A, Ohm R, Otillar R . MycoCosm portal: gearing up for 1000 fungal genomes. Nucleic Acids Res. 2013; 42(Database issue):D699-704. PMC: 3965089. DOI: 10.1093/nar/gkt1183. View

20.

Schmidt M, Hamacher K, Reinhardt F, Lotz T, Groher F, Suess B . SICOR: Subgraph Isomorphism Comparison of RNA Secondary Structures. IEEE/ACM Trans Comput Biol Bioinform. 2019; 17(6):2189-2195. DOI: 10.1109/TCBB.2019.2926711. View