Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding

Modeling textual or visual information with vector representations trained from large language or visual datasets has been successfully explored in recent years. However, tasks such as visual question answering require combining these vector representations with each other. Approaches to multimodal pooling include element-wise product or sum, as well as concatenation of the visual and textual representations. We hypothesize that these methods are not as expressive as an outer product of the visual and textual vectors. As the outer product is typically infeasible due to its high dimensionality, we instead propose utilizing Multimodal Compact Bilinear pooling (MCB) to efficiently and expressively combine…

• NS
  National Science Foundation

  Awards: IIS-1212798, IIS-1427425, 1427425, 1212798, N000141110688
• DA
  Defense Advanced Research Projects Agency
• MU
  Multidisciplinary University Research Initiative

  Award: N000141110688
• AF
  Air Force Research Laboratory