Classifying Emotions and Engagement in Online Learning Based on a Single Facial Expression Recognition Neural Network

In this article, behaviour of students in the e-learning environment is analyzed. The novel pipeline is proposed based on video facial processing. At first, face detection, tracking and clustering techniques are applied to extract the sequences of faces of each student. Next, a single efficient neural network is used to extract emotional features in each frame. This network is pre-trained on face identification and fine-tuned for facial expression recognition on static images from AffectNet using a specially developed robust optimization technique. It is shown that the resulting facial features can be used for fast simultaneous prediction of students’ engagement levels (from disengaged to highly engaged),…

• AC
  Analytical Center for the Government of the Russian Federation

  Award: 0D730321P5Q0002
• NR
  National Research University Higher School of Economics

  Award: 70-2021-00139