VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking

Scale is the primary factor for building a powerful foundation model that could well generalize to a variety of downstream tasks. However, it is still challenging to train video foundation models with billions of parameters. This paper shows that video masked autoencoder (VideoMAE) is a scalable and general self-supervised pre-trainer for building video foundation models. We scale the VideoMAE in both model and data with a core design. Specifically, we present a dual masking strategy for efficient pre-training, with an encoder operating on a subset of video tokens and a decoder processing another subset of video tokens. Although VideoMAE is very efficient due to high masking ratio in encoder, masking decoder…

• NN
  National Natural Science Foundation of China

  Award: 62076119,61921006
• YI
  Youth Innovation Promotion Association of the Chinese Academy of Sciences

  Award: 2020355
• NK
  National Key Research and Development Program of China

  Award: 2022ZD0160900,2022ZD0160100