Comparing Recognition Performance and Robustness of Multimodal Deep Learning Models for Multimodal Emotion Recognition

Multimodal signals are powerful for emotion recognition since they can represent emotions comprehensively. In this article, we compare the recognition performance and robustness of two multimodal emotion recognition models: 1) deep canonical correlation analysis (DCCA) and 2) bimodal deep autoencoder (BDAE). The contributions of this article are threefold: 1) we propose two methods for extending the original DCCA model for multimodal fusion: a) weighted sum fusion and b) attention-based fusion; 2) we systemically compare the performance of DCCA, BDAE, and traditional approaches on five multimodal data sets; and 3) we investigate the robustness of DCCA, BDAE, and traditional approaches on SEED-V and DREAMER…

• NN
  National Natural Science Foundation of China

  Award: 61976135
• CS
  China Southern Power Grid

  Award: GDKJXM20185761
• HE
  Higher Education Discipline Innovation Project
• NK
  National Key Research and Development Program of China

  Award: 2017YFB1002501
• FR
  Fundamental Research Funds for the Central Universities