Visual Prompt Multi-Modal Tracking

Visible-modal object tracking gives rise to a series of downstream multi-modal tracking tributaries. To inherit the powerful representations of the foundation model, a natural modus operandi for multi-modal tracking is full fine-tuning on the RGB-based parameters. Albeit effective, this manner is not optimal due to the scarcity of downstream data and poor transferability, etc. In this paper, inspired by the recent success of the prompt learning in language models, we develop Visual Prompt multi-modal Tracking (ViPT), which learns the modal-relevant prompts to adapt the frozen pre-trained foundation model to various downstream multi-modal tracking tasks. ViPT finds a better way to stimulate the knowledge of the…

• NN
  National Natural Science Foundation of China

  Award: 62293542,U1903215
• NK
  National Key Research and Development Program of China

  Award: 2018AAA0102001
• FR
  Fundamental Research Funds for the Central Universities

  Award: DUT22ZD210