PPT: Pre-trained Prompt Tuning for Few-shot Learning
Center for Information Technology · Tsinghua University · +1 more institution
Abstract
Prompts for pre-trained language models (PLMs) have shown remarkable performance by bridging the gap between pre-training tasks and various downstream tasks. Among these methods, prompt tuning, which freezes PLMs and only tunes soft prompts, provides an efficient and effective solution for adapting largescale PLMs to downstream tasks. However, prompt tuning is yet to be fully explored. In our pilot experiments, we find that prompt tuning performs comparably with conventional full-model tuning when downstream data are sufficient, whereas it is much worse under fewshot learning settings, which may hinder the application of prompt tuning. We attribute this low performance to the manner of initializing soft…
Citation impact
- FWCI
- 24.10
- Percentile
- 100%
- References
- 58
Authors
4- YGYuxian Gu
Center for Information Technology, Tsinghua University
- XHXu Han
Center for Information Technology, Tsinghua University
- ZLZhiyuan Liu
Center for Information Technology, Beijing Academy of Artificial Intelligence, Tsinghua University
- MHMinlie HuangCorresponding
Center for Information Technology, Beijing Academy of Artificial Intelligence, Tsinghua University
Topics & keywords
- Computer science
- Initialization
- Task (project management)
- Bridging (networking)
- Artificial intelligence
- Generalization
- Machine learning