AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Irvine University · University of California, Irvine · +1 more institution
Abstract
The remarkable success of pretrained language models has motivated the study of what kinds of knowledge these models learn during pretraining. Reformulating tasks as fillin-the-blanks problems (e.g., cloze tests) is a natural approach for gauging such knowledge, however, its usage is limited by the manual effort and guesswork required to write suitable prompts. To address this, we develop AUTOPROMPT, an automated method to create prompts for a diverse set of tasks, based on a gradient-guided search. Using AUTO-PROMPT, we show that masked language models (MLMs) have an inherent capability to perform sentiment analysis and natural language inference without additional parameters or finetuning, sometimes…
Citation impact
- FWCI
- 79.70
- Percentile
- 100%
- References
- 31
Authors
5- TSTaylor ShinCorresponding
Irvine University, University of California, Irvine, University of California, Berkeley
- YRYasaman Razeghi
Irvine University, University of California, Irvine, University of California, Berkeley
- RLRobert L. Logan
Irvine University, University of California, Irvine, University of California, Berkeley
- EWEric Wallace
Irvine University, University of California, Irvine, University of California, Berkeley
- SSSameer Singh
University of California, Berkeley, University of California, Irvine, Irvine University
Topics & keywords
- Computer science
- Benchmark (surveying)
- Artificial intelligence
- Relation (database)
- Set (abstract data type)
- Inference
- Natural language processing
- Language model
- Quality Education