Integrating Multimodal Information in Large Pretrained Transformers

Recent Transformer-based contextual word representations, including BERT and XLNet, have shown state-of-the-art performance in multiple disciplines within NLP. Fine-tuning the trained contextual models on task-specific datasets has been the key to achieving superior performance downstream. While fine-tuning these pre-trained models is straight-forward for lexical applications (applications with only language modality), it is not trivial for multimodal language (a growing area in NLP focused on modeling face-to-face communication). Pre-trained models don't have the necessary components to accept two extra modalities of vision and acoustic. In this paper, we proposed an attachment to BERT and XLNet called…

• NS
  National Science Foundation

  Awards: 1750439, 1722822, W911NF-19
• NI
  National Institutes of Health
• DA
  Defense Advanced Research Projects Agency

  Awards: W911NF-19-1-0029, W911NF-15-1-0542
• AR
  Advanced Research Projects Agency
• AR
  Army Research Office

  Awards: W911NF-19-1, W911NF-15-1-, W911NF, W911NF-19