Visual-Language Prompt Tuning with Knowledge-Guided Context Optimization
Artificial Intelligence in Medicine (Canada) · Shandong Institute of Automation · +2 more institutions
Abstract
Prompt tuning is an effective way to adapt the pretrained visual-language model (VLM) to the downstream task using task-related textual tokens. Representative CoOp-based work combines the learnable textual tokens with the class tokens to obtain specific textual knowledge. However, the specific textual knowledge is worse generalization to the unseen classes because it forgets the essential general textual knowledge having a strong generalization ability. To tackle this issue, we introduce a novel Knowledge-guided Context Optimization (KgCoOp) to enhance the generalization ability of the learnable prompt for unseen classes. The key insight of KgCoOp is that the forgetting about essential knowledge can be…
Citation impact
- FWCI
- 22.55
- Percentile
- 100%
- References
- 60
Authors
3Topics & keywords
- Computer science
- Generalization
- Discriminative model
- Task (project management)
- Artificial intelligence
- Context (archaeology)
- Forgetting
- Natural language processing
- Reduced inequalities