Bilingual language model for protein sequence and structure

Abstract Adapting language models to protein sequences spawned the development of powerful protein language models (pLMs). Concurrently, AlphaFold2 broke through in protein structure prediction. Now we can systematically and comprehensively explore the dual nature of proteins that act and exist as three-dimensional (3D) machines and evolve as linear strings of one-dimensional (1D) sequences. Here, we leverage pLMs to simultaneously model both modalities in a single model. We encode protein structures as token sequences using the 3Di-alphabet introduced by the 3D-alignment method Foldseek. For training, we built a non-redundant dataset from AlphaFoldDB and fine-tuned an existing pLM (ProtT5) to translate…

• AV
  Alexander von Humboldt-Stiftung
• NR
  National Research Foundation

  Award: 2021-R1C1-C102065
• DF
  Deutsche Forschungsgemeinschaft

  Award: DFG-GZ: RO1320/4-1
• BF
  Bundesministerium für Bildung und Forschung
• SN
  Seoul National University
• NR
  National Research Foundation of Korea

  Awards: 2021-R1C1-C102065, 2021-M3A9-I4021220, RS-2023-00250470, 2020M3-A9G7-103933
• TU
  Technische Universität München
• S
  Samsung