Fine-tuning protein language models boosts predictions across diverse tasks

Prediction methods inputting embeddings from protein language models have reached or even surpassed state-of-the-art performance on many protein prediction tasks. In natural language processing fine-tuning large language models has become the de facto standard. In contrast, most protein language model-based protein predictions do not back-propagate to the language model. Here, we compare the fine-tuning of three state-of-the-art models (ESM2, ProtT5, Ankh) on eight different tasks. Two results stand out. Firstly, task-specific supervised fine-tuning almost always improves downstream predictions. Secondly, parameter-efficient fine-tuning can reach similar improvements consuming substantially fewer resources at…

• AV
  Alexander von Humboldt-Stiftung
• DF
  Deutsche Forschungsgemeinschaft

  Awards: DFG-GZ: RO1320/4-1, RO1320/4-1
• BF
  Bundesministerium für Bildung und Forschung
• TU
  Technische Universität München
• AD
  AbbVie Deutschland