Context Encoders: Feature Learning by Inpainting

We present an unsupervised visual feature learning algorithm driven by context-based pixel prediction. By analogy with auto-encoders, we propose Context Encoders - a convolutional neural network trained to generate the contents of an arbitrary image region conditioned on its surroundings. In order to succeed at this task, context encoders need to both understand the content of the entire image, as well as produce a plausible hypothesis for the missing part(s). When training context encoders, we have experimented with both a standard pixel-wise reconstruction loss, as well as a reconstruction plus an adversarial loss. The latter produces much sharper results because it can better handle multiple modes in the…

• NS
  National Science Foundation

  Awards: IIS-1212798, IIS-1427425, 1411106, 1427425, 1212798, 1536003, N000141110688, 0001411
• DA
  Defense Advanced Research Projects Agency
• MU
  Multidisciplinary University Research Initiative

  Award: N000141110688
• AF
  Air Force Research Laboratory