Abstract:

Automatic human emotion recognition has been attracting a lot of efforts that led to significant results in improving human-computer interaction. Out of various ways to detect human emotions, the study of facial expressions provides vital information about the underlying emotion, thus being a constantly growing field of research with connection to computer vision, artificial intelligence, and computer engineering.

With the current restrictions on face-to-face interaction between teachers and learners, the necessity of conducting academic activities in online environments has become stringent. Since individuals are so different in matters of acting and thinking, it is obvious that the mindset adopted by a student during times of high-stress levels also differs. Moreover, online courses and e-learning activities that needed to take place during these rough times of pandemic changed the innate teaching flow of both professors and students, which increased to the already existent stress level. The educational actors need to adapt to the online environment, get accustomed to other learning tools and methods, and find new ways of evaluation and being evaluated.

This is where online student emotion recognition could become a key factor in understanding how all these sudden changes affected both professors and students. It can become a powerful tool to measure emotions felt by students during online courses, where the distance can be considered a drawback in the learning and understanding process. It can also inform the refinement and development of new teaching strategies that are adapted to these new conditions.

This paper proposes a model of facial emotion recognition based on the analysis of a student’s face from photos and real-time video and the detection of the main characteristics for ten basic emotions, namely: anger, contempt, disgust, embarrassment, fear, joy, pride, sadness, surprise, and the neutral state. The target group for this application is represented by students that are engaged in online learning activities, whether they are lectures or practical activities.

The application was developed using Matlab and makes use of its various image processing libraries. Our application uses the webcam for acquiring photos and videos which are further divided into frames. Each of these frames is firstly pre-processed, meaning the relevant regions for the emotion recognition process are cropped. Next, the local binary patterns (LBP) and histograms of oriented gradients (HOG) features are being extracted from them. These features are then used in the classification (which was based on the k-nearest neighbors (KNN) algorithm) and prediction processes. The resulted emotion is finally displayed in an annotated shape that marks the facial region.

The paper concludes with formulating some directions for further software developments, including a discussion of the limitations and possible improvements of the current version.

Keywords:

Human-computer interaction, e-learning, student emotion, real-time, computer vision, histograms of oriented gradients, local binary patterns.