RMT: Retentive Networks Meet Vision Transformers

Vision Transformer (ViT) has gained increasing attention in the computer vision community in recent years. How-ever, the core component of ViT, Self-Attention, lacks ex-plicit spatial priors and bears a quadratic computational complexity, thereby constraining the applicability of ViT. To alleviate these issues, we draw inspiration from the re-cent Retentive Network (RetNet) in the field of NLP, and propose RMT, a strong vision backbone with explicit spa-tial prior for general purposes. Specifically, we extend the RetNet's temporal decay mechanism to the spatial do-main, and propose a spatial decay matrix based on the Manhattan distance to introduce the explicit spatial prior to Self-Attention. Additionally, an…

• NN
  National Natural Science Foundation of China

  Award: U21B2045
• YI
  Youth Innovation Promotion Association of the Chinese Academy of Sciences

  Award: 2022132
• BN
  Beijing Nova Program

  Award: 20230484276