Parameter-efficient fine-tuning of large-scale pre-trained language models

Abstract With the prevalence of pre-trained language models (PLMs) and the pre-training–fine-tuning paradigm, it has been continuously shown that larger models tend to yield better performance. However, as PLMs scale up, fine-tuning and storing all the parameters is prohibitively costly and eventually becomes practically infeasible. This necessitates a new branch of research focusing on the parameter-efficient adaptation of PLMs, which optimizes a small portion of the model parameters while keeping the rest fixed, drastically cutting down computation and storage costs. In general, it demonstrates that large-scale models could be effectively stimulated by the optimization of a few parameters. Despite the…

• NN
  National Natural Science Foundation of China

  Award: 62276154
• TU
  Tsinghua University
• NK
  National Key Research and Development Program of China

  Award: 2020AAA0106500