MetaFormer Baselines for Vision

MetaFormer, the abstracted architecture of Transformer, has been found to play a significant role in achieving competitive performance. In this paper, we further explore the capacity of MetaFormer, again, by migrating our focus away from the token mixer design: we introduce several baseline models under MetaFormer using the most basic or common mixers, and demonstrate their gratifying performance. We summarize our observations as follows: (1) MetaFormer ensures solid lower bound of performance. By merely adopting identity mapping as the token mixer, the MetaFormer model, termed IdentityFormer, achieves [Formula: see text]80% accuracy on ImageNet-1 K. (2) MetaFormer works well with arbitrary token mixers. When…

• NU
  National University of Singapore

  Award: A-0005947-21-00
• NR
  National Research Foundation Singapore

  Award: AISG2-RP-2021-023
• AR
  Advanced Research and Technology Innovation Centre, College of Design and Engineering, National University of Singapore