Multimodal Emotion Recognition

New demands associated with living in a highly-technological and globally-competitive world require today's students to develop a very different set of competencies than previous generations of students needed. The general goal of education is to prepare young people to live independent and productive lives. Unfortunately, our current educational system is not keeping pace with these changes and new demands. The world is becoming increasingly complex and to make progress toward fixing educational woes, we need to have a good sense of bearings-where we are, and where we're heading. This paper is intended to provide such bearings, specifically in terms of a fresh vision for education. We envision new modes of learning and teaching using stimulating online environments such as games and simulations, coupled with an assessment infrastructure covering a broad set of competencies and other attributes to support learning. This represents a long view of the field to inform current R&D efforts.

Multimodal emotion recognition has recently gained much attention since it can leverage diverse and complementary relationships over multiple modalities, such as audio, visual, and biosignals. Most state-of-the-art methods for audio-visual (A-V) fusion rely on recurrent networks or conventional attention mechanisms that do not effectively leverage the complementary nature of A-V modalities. In this paper, we focus on dimensional emotion recognition based on the fusion of facial and vocal modalities extracted from videos. Specifically, we propose a joint cross-attention model that relies on the complementary relationships to extract the salient features across A-V modalities, allowing for accurate prediction of continuous values of valence and arousal. The proposed fusion model efficiently leverages the inter-modal relationships, while reducing the heterogeneity between features. In particular, it computes cross-attention weights based on the correlation between joint feature representations, and that of individual modalities. By deploying a joint A-V feature representation into the crossattention module, the performance of our fusion module improves significantly over the vanilla cross-attention module. Experimental results 1 on the AffWild2 dataset highlight the robustness of our proposed A-V fusion model. It has achieved a concordance correlation coefficient (CCC) of 0.374 (0.663) and 0.363 (0.584) for valence and arousal, respectively, on test set (validation set). This is a significant improvement over the baseline of third challenge of Affective Behavior Analysis in-the-wild (ABAW3) competition, with a CCC of 0.180 (0.310) and 0.170 (0.170).

New demands associated with living in a highly-technological and globally-competitive world require today's students to develop a very different set of competencies than previous generations of students needed. The general goal of education is to prepare young people to live independent and productive lives. Unfortunately, our current educational system is not keeping pace with these changes and new demands. The world is becoming increasingly complex and to make progress toward fixing educational woes, we need to have a good sense of bearings-where we are, and where we're heading. This paper is intended to provide such bearings, specifically in terms of a fresh vision for education. We envision new modes of learning and teaching using stimulating online environments such as games and simulations, coupled with an assessment infrastructure covering a broad set of competencies and other attributes to support learning. This represents a long view of the field to inform current R&D efforts.

The paper presents our initial attempts in building an audio video emotion recognition system. Both, audio and video subsystems are discussed, and description of the database of spontaneous emotions is given. The task of labelling the recordings from the database according to different emotions is discussed and the measured agreement between multiple annotators is presented. Instead of focusing on the prosody in audio emotion recognition, we evaluate the possibility of using linear transformations (CMLLR) as features. The classification results from audio and video subsystems are combined using sum rule fusion and the increase in recognition results, when using both modalities, is presented.

We describe our system for empathic emotion recognition. It is based on deep learning on multiple modalities in a late fusion architecture. We describe the modules of our system and discuss the evaluation results. Our code is also available for the research community 1

For many years, the emotion recognition task has remained one of the most interesting and important problems in the field of human-computer interaction. In this study, we consider the emotion recognition task as a classification as well as a regression task by processing encoded emotions in different datasets using deep learning models. Our model combines convolutional neural network (CNN) with recurrent neural network (RNN) to predict dimensional emotions on video data. At the first step, CNN extracts feature vectors from video frames. In the second step, we fed these feature vectors to train RNN for exploiting the temporal dynamics of video. Furthermore, we analyzed how each neural network contributes to the systems overall performance. The experiments are performed on publicly available datasets including the largest modern Aff-Wild2 database. It contains over sixty hours of video data. We discovered the problem of overfitting of the model on an unbalanced dataset with an illustrative example using confusion matrices. The problem is solved by downsampling technique to balance the dataset. By significantly decreasing training data, we balance the dataset, thereby, the overall performance of the model is improved. Hence, the study qualitatively describes the abilities of deep learning models exploring enough amount of data to predict facial emotions. Our proposed method is implemented using Tensorflow Keras. The code is publicly available in repository 1 .

In recent years, the main problem in e-learning has shifted from analyzing content to personalization of learning environment by Intelligence Tutoring Systems (ITSs). Therefore, by designing personalized teaching models, learners are able to have a successful and satisfying experience in achieving their learning goals. Affective Tutoring Systems (ATSs) are some kinds of ITS that can recognize and respond to affective states of learners. In this study, we designed, implemented, and evaluated a system to personalize the learning environment based on the facial emotions recognition, head pose estimation, and cognitive style of learners. First, a unit called Intelligent Analyzer (AI) created which was responsible for recognizing facial expression and head angles of learners. Next, the ATS was built which mainly made of two units: ITS, IA. Results indicated that with the ATS, participants needed less efforts to pass the tests. In other words, we observed when the IA unit was activated, learners could pass the final tests in fewer attempts than those for whom the IA unit was deactivated. Additionally, they showed an improvement in terms of the mean passing score and academic satisfaction.

The paper presents our initial attempts in building an audio video emotion recognition system. Both, audio and video subsystems are discussed, and description of the database of spontaneous emotions is given. The task of labelling the recordings from the database according to different emotions is discussed and the measured agreement between multiple annotators is presented. Instead of focusing on the prosody in audio emotion recognition, we evaluate the possibility of using linear transformations (CMLLR) as features. The classification results from audio and video subsystems are combined using sum rule fusion and the increase in recognition results, when using both modalities, is presented.

Introduction: Psychotherapists' emotional and empathic competencies have a positive influence on psychotherapy outcome and alliance. However, it is doubtful whether psychotherapy education in itself leads to improvements in trainee psychotherapists' emotion recognition accuracy (ERA), which is an essential part of these competencies. Methods: In a randomized, controlled, double-blind study (N = 68), we trained trainee psychotherapists (57% psychodynamic therapy and 43% cognitive behavioral therapy) to detect non-verbal emotional expressions in others using standardized computerized trainings-one for multimodal emotion recognition accuracy and one for micro expression recognition accuracy-and compared their results to an active control group one week after the training (n = 60) and at the one-year follow up (n = 55). The participants trained once weekly during a three-week period. As outcome measures, we used a multimodal emotion recognition accuracy task, a micro expression recognition accuracy task and an emotion recognition accuracy task for verbal and non-verbal (combined) emotional expressions in medical settings. Results: The results of mixed multilevel analyses suggest that the multimodal emotion recognition accuracy training led to significantly steeper increases than the other two conditions from pretest to the posttest one week after the last training session. When comparing the pretest to follow-up differences in slopes, the superiority of the multimodal training group was still detectable in the unimodal audio modality and the unimodal video modality (in comparison to the control training group), but not when considering the multimodal audio-video modality or the total score of the multimodal emotion recognition accuracy measure. The micro expression training group showed a significantly steeper change trajectory from pretest to posttest compared to the control training group, but not compared to the multimodal training group. However, the effect vanished again until the one-year follow-up. There were no differences in change trajectories for the outcome measure about emotion recognition accuracy in medical settings. Discussion: We conclude that trainee psychotherapists' emotion recognition accuracy can be effectively trained, especially multimodal emotion recognition accuracy, and suggest that the changes in unimodal emotion recognition accuracy (audio-only and video-only) are long-lasting. Implications of these findings for the psychotherapy education are discussed.

In this work we tackle the task of video-based audiovisual emotion recognition, within the premises of the 2nd Workshop and Competition on Affective Behavior Analysis in-the-wild (ABAW2). Poor illumination conditions, head/body orientation and low image resolution constitute factors that can potentially hinder performance in case of methodologies that solely rely on the extraction and analysis of facial features. In order to alleviate this problem, we leverage both bodily and contextual features, as part of a broader emotion recognition framework. We choose to use a standard CNN-RNN cascade as the backbone of our proposed model for sequence-to-sequence (seq2seq) learning. Apart from learning through the RGB input modality, we construct an aural stream which operates on sequences of extracted mel-spectrograms. Our extensive experiments on the challenging and newly assembled Aff-Wild2 dataset verify the validity of our intuitive multi-stream and multi-modal approach towards emotion recognition "in-the-wild". Emphasis is being laid on the the beneficial influence of the human body and scene context, as aspects of the emotion recognition process that have been left relatively unexplored up to this point. All the code was implemented using PyTorch 1 and is publicly available 2 .

Emotion is assuming increasing importance in human computer interaction (HCI), in general, with the growing feeling that emotion is central to human communication and intelligence. Users expect not just functionality as a factor of usability, but experiences, matched to their expectations, emotional states, and interaction goals. Endowing computers with this kind of intelligence for HCI is a complex task. It becomes more complex with the fact that the interaction of humans with their environment (including other humans) is naturally multimodal. In reality, one uses a combination of modalities and they are not treated independently. In an attempt to render HCI more similar to human-human communication and enhance its naturalness, research on multiple modalities of human expressions has seen ongoing progress in the past few years. As compared to unimodal approaches, various problems arise in case of multimodal emotion recognition especially concerning fusion architecture of multimodal information. In this paper we will be proposing a rule based hybrid approach to combine the information from various sources for recognizing the target emotions. The results presented in this paper shows that it is feasible to recognize human affective states with a reasonable accuracy by combining the modalities together using rule based system.

In day to day life, emotions are becoming an important tool which helps to take not only the decisions but also to enhance learning, creative thinking and to effectively correspond in the social interaction. Several studies have been conducted comprising of classical human human interaction and human computer interaction. They concluded that for intelligent interaction, emotions play an important role. By embedding the emotions in the interaction of human with machine, machine would be in a position to sense the mood of the user and change its interaction accordingly. Hence the system will be friendlier to the user and its responses will be more similar to human behaviour. In general, human beings make use of emotions through speech, facial expression and gestures for conveying the crucial information. This paper presents an attempt to recognize selected emotion categories from keyboard stroke pattern. The emotional categories considered for our analysis are neutral, positive and negative. We

Emotion recognition can be used in wide range of applications, we are interested in E-learning system because of several benefits of learning anywhere anyplace and at anytime . The vision of affective computing is to impart to system, the possibility of understanding the user through recognizing his emotions and expressions. Obviously, the entire emotional state of a user is expressed and can be observed in different modalities. In this paper, a survey was conducted on the latest research in emotion recognition, first with unimodal system then with multimodal system, specifically applied in E-learning environment. Then, we propose the improvement of EMASPEL (Emotional Multi-Agents System for Peer-to-peer E-Learning) by the fusion of several realtime modalities of emotional communication.

Web-based education is particularly appropriate for remote teaching and learning at any time and place, away from classrooms and does not necessarily require the presence of a human instructor. The need for time and place independence is even greater in some cases, such as for medical instructors who are usually doctors that have to treat patients on top of their tutoring duties. However, this independence from real teachers and classrooms may influence negatively the students who may feel deprived of the benefits of human-human interaction. In this paper we describe a novel approach for incorporating affective characteristics into e-learning through an authoring tool. The authoring tool incorporates and adapts principles of a cognitive theory for modeling possible emotional states that a tutoring agent may use for educational purposes. Medical instructors may use this authoring tool to create their own educational characters that will interact affectively with their students in the e-learning environment.

Facial behavior analysis is a broad topic with various categories such as facial emotion recognition, age, and gender recognition. Many studies focus on individual tasks while the multi-task learning approach is still an open research issue and requires more research. In this paper, we present our solution and experiment result for the Multi-Task Learning challenge of the Affective Behavior Analysis in-the-wild competition. The challenge is a combination of three tasks: action unit detection, facial expression recognition, and valance-arousal estimation. To address this challenge, we introduce a cross-attentive module to improve multi-task learning performance. Additionally, a facial graph is applied to capture the association among action units. As a result, we achieve the evaluation measure of 128.8 on the validation data provided by the organizers, which outperforms the baseline result of 30.

Affective computing has gained significant attention from researchers in the last decade due to the wide variety of applications that can benefit from this technology. Often, researchers describe affect using emotional dimensions such as arousal and valence. Valence refers to the spectrum of negative to positive emotions while arousal determines the level of excitement. Describing emotions through continuous dimensions (e.g. valence and arousal) allows us to encode subtle and complex affects as opposed to discrete emotions, such as the basic six emotions: happy, anger, fear, disgust, sad and neutral. Recognizing spontaneous and subtle emotions remains a challenging problem for computers. In our work, we employ two modalities of information: video and audio. Hence, we extract visual and audio features using deep neural network models. Given that emotions are time-dependent, we apply the Temporal Convolutional Neural Network (TCN) to model the variations in emotions. Additionally, we investigate an alternative model that combines a Convolutional Neural Network (CNN) and a Recurrent Neural Network (RNN). Given our inability to fit the latter deep model into the main memory, we divide the RNN into smaller segments and propose a scheme to backpropagate gradients across all segments. We configure the hyperparameters of all models using Gaussian processes to obtain a fair comparison between the proposed models. Our results show that TCN outperforms RNN for the recognition of the arousal and valence emotional dimensions. Therefore, we propose the adoption of TCN for emotion detection problems as a baseline method for future work. Our experimental results show that TCN outperforms all RNN based models yielding a concordance correlation coefficient of 0.7895 (vs. 0.7544) on valence and 0.8207 (vs. 0.7357) on arousal on the validation dataset of SEWA dataset for emotion prediction.

Our work aims at helping teachers to monitor learners' engagement during mind mapping activities. The use of mind maps helps students to elaborate cognitive learning strategies like creating and organizing contents. To assess the quality of these mind maps, teachers need tools to understand students' choices and strategies when constructing their mind maps. We have defined in collaboration with teachers a set of indicators based on learners' interaction traces. Those indicators have been implemented and integrated in a dashboard dedicated to teachers. In this paper, we introduce a first version of the dashboard: its design, implementation, and the results of its evaluation conducted during interviews with real teachers.

Learning analytic dashboards (LADs) are data visualization systems that use dynamic data in digital learning environments to provide students, teachers, and administrators with a wealth of information about student’s engagement, experiences, and performance on tasks. LADs have become increasingly popular, particularly in formative learning contexts, and help teachers make data-informed decisions about a student’s developing skills on a topic. LADs afford the possibility for teachers to obtain real-time data on student performance, response processes, and progress on academic learning tasks. However, data presented on LADs are often not based on an evaluation of stakeholder needs, and have been found to not be clearly interpretable and actionable for teachers to readily adapt their pedagogical actions based on these insights. We elaborate on how insights from research focused on interpretation and use of Score Reporting systems and research on open learner models (OLMs) can be used t...

Nonverbal emotion recognition accuracy (ERA) is a central feature of successful communication and interaction, and is of importance for many professions. We developed and evaluated two ERA training programs—one focusing on dynamic multimodal expressions (audio, video, audio-video) and one focusing on facial micro expressions. Sixty-seven subjects were randomized to one of two experimental groups (multimodal, micro expression) or an active control group (emotional working memory task). Participants trained once weekly with a brief computerized training program for three consecutive weeks. Pre-post outcome measures consisted of a multimodal ERA task, a micro expression recognition task, and a task about patients' emotional cues. Post measurement took place approximately a week after the last training session. Non-parametric mixed analyses of variance using the Aligned Rank Transform were used to evaluate the effectiveness of the training programs. Results showed that multimodal tr...

This paper represents our newly developed software for emotion recognition from facial expressions. Besides allowing emotion recognition from image files and recorded video files, it uses webcam data to provide real-time, continuous, and unobtrusive facial emotional expressions. It uses FURIA algorithm for unordered fuzzy rule induction to offer timely and appropriate feedback based on learners' facial expressions. The main objective of this study was first to validate the use of webcam data for a real-time and accurate analysis of facial expressions in elearning environments. Second, transform these facial expressions to detected emotional states using the FURIA algorithm. We measured the performance of the software with ten participants, provided them with the same computer-based tasks, requested them a hundred times to mimic specific facial expressions, and recorded all sessions on video. We used the recorded video files to feed our newly developed software. We then used two experts' opinions to annotate and rate participants' recorded behaviours and to validate the software's results. The software provides accurate and reliable results with the overall accuracy of 83.2%, which is comparable to the recognition by humans. This study will help to increase the quality of e-learning.

The teaching of communication skills is a labour-intensive task because of the detailed feedback that should be given to learners during their prolonged practice. This study investigates to what extent our FILTWAM facial and vocal emotion recognition software can be used for improving a serious game (the Communication Advisor) that delivers a web-based training of communication skills. A test group of 25 participants played the game wherein they were requested to mimic specific facial and vocal emotions. Half of the assignments included direct feedback and the other half included no feedback. It was investigated whether feedback on the mimicked emotions would lead to better learning. The results suggest the facial performance growth was found to be positive, particularly significant in the feedback condition. The vocal performance growth was significant in both conditions. The results are a significant indication that the automated feedback from the software improves learners' communication performances.

This article describes the validation study of our software that uses combined webcam and microphone data for real-time, continuous, unobtrusive emotion recognition as part of our FILTWAM framework. FILTWAM aims at deploying a real-time multimodal emotion recognition method for providing more adequate feedback to the learners through an online communication skills training. Herein, timely feedback is needed that reflects on the intended emotions they show and which is also useful to increase learners' awareness of their own behavior. At least, a reliable and valid software interpretation of performed face and voice emotions is needed to warrant such adequate feedback. This validation study therefore calibrates our software. The study uses a multimodal fusion method. Twelve test persons performed computer-based tasks in which they were asked to mimic specific facial and vocal emotions. All test persons' behavior was recorded on video and two raters independently scored the showed emotions, which were contrasted with the software recognition outcomes. A hybrid method for multimodal fusion of our multimodal software shows accuracy between 96.1% and 98.6% for the best-chosen WEKA classifiers over predicted emotions. The software fulfils its requirements of real-time data interpretation and reliable results.

This paper presents the voice emotion recognition part of the FILTWAM framework for real-time emotion recognition in affective e-learning settings. FILT WAM (Framework for Improving Learning Through Webcams And Microphones) intends to offer timely and appropriate online feedback based upon learner's vocal intonations and facial expressions in order to foster their learning. Whereas the facial emotion recognition part has been successfully tested in a previous study, the here presented study describes the development and testing of FILTWAM's vocal emotion recognition software artefact. The main goal of this study was to show the valid use of computer microphone data for real-time and adequate interpretation of vocal intonations into extracted emotional states. The software that was developed was tested in a study with 12 participants. All participants individually received the same computerbased tasks in which they were requested 80 times to mimic specific vocal expressions (960 occurrences in total). Each individual session was recorded on video. For the validation of the voice emotion recognition software artefact, two experts annotated and rated participants' recorded behaviours. Expert findings were then compared with the software recognition results and showed an overall accuracy of Kappa of 0.743. The overall accuracy of the voice emotion recognition software artefact is 67 % based on the requested emotions and the recognized emotions. Our FILTWAM-software allows to continually and unobtrusively observing learners' behaviours and transforms these behaviours into emotional states. This paves the way for unobtrusive and real-time capturing of learners' emotional states for enhancing adaptive e-learning approaches.

This paper presents a framework (FILTWAM) for real time emotion recognition in e-learning by using webcams. FILTWAM (Framework for Improving Learning Through Webcams And Microphones) offers timely and relevant feedback based upon learner's facial expressions and verbalizations. FILTWAM's facial expression software module has been developed and tested in a proof of concept study. The main goal of this study was to validate the use of webcam data for a real-time and adequate interpretation of facial expressions into extracted emotional states. The software was calibrated with ten test persons. They received the same computer-based tasks in which each of them were requested a hundred times to mimic specific facial expressions. All sessions were recorded on video. For the validation of the face emotion recognition software, two experts annotated and rated participants' recorded behaviours. Expert findings were contrasted with the software results and showed an overall value of Kappa of 0.77. An overall accuracy of our software based on the requested emotions and the recognized emotions is 72%. Whereas existing software only allows not-real time, discontinuous and obtrusive facial detection, our software allows to continuously and unobtrusively monitor learners' behaviours and converts these behaviours directly into emotional states. This paves the way for enhancing the quality and efficacy of e-learning by including the learner's emotional states.

This paper introduces the voice emotion recognition part of our framework for improving learning through webcams and microphones (FILTWAM). This framework enables multimodal emotion recognition of learners during game-based learning. The main goal of this study is to validate the use of microphone data for a real-time and adequate interpretation of vocal expressions into emotional states were the software is calibrated with end users. FILTWAM already incorporates a valid face emotion recognition module and is extended with a voice emotion recognition module. This extension aims to provide relevant and timely feedback based upon learner's vocal intonations. The feedback is expected to enhance learner's awareness of his or her own behavior. Six test persons received the same computer-based tasks in which they were requested to mimic specific vocal expressions. Each test person mimicked 82 emotions, which led to a dataset of 492 emotions. All sessions were recorded on video. An overall accuracy of our software based on the requested emotions and the recognized emotions is a pretty good 74.6% for the emotions happy and neutral emotions; but will be improved for the lower values of an extended set of emotions. In contrast with existing software our solution allows to continuously and unobtrusively monitor learners' intonations and convert these intonations into emotional states. This paves the way for enhancing the quality and efficacy of game-based learning by including the learner's emotional states, and links these to pedagogical scaffolding.

This paper introduces a Framework for Improving Learning Through Webcams And Microphones (FILTWAM). It proposes an overarching framework comprising conceptual and technical frameworks for enhancing the online communication skills of lifelong learners. Our approach interprets the emotional state of people using webcams and microphones and combines relevant and timely feedback based upon learner's facial expressions and verbalizations (like sadness, anger, disgust, fear, happiness, surprise, and neutral). The feedback generated from the webcams is expected to enhance learner's awareness of their own behavior. Our research enhances flexibility and scalability in contrast with face-to-face trainings and better helps the interests of lifelong learners who prefer to study at their own pace, place and time. Our small-scale proof of concept study exemplifies the practical application of FILTWAM and provides first evaluation results on that. This study will guide future development of software, training materials, and research. It will validate the use of webcam data for a real-time and adequate interpretation of facial expressions into emotional states. Participants' behaviour is recorded on videos so that videos will be replayed, rated, annotated and evaluated by expert observers and contrasted with participants' own opinions in future research.

Stealth assessment derives the progression of learning in an unobtrusive way from observed gameplay captured in log files. To this end, it uses machine learning technologies to provide probabilistic reasoning over established latent competency variable models. Now that video games are increasingly being used for training and learning purposes, stealth assessment could provide an excellent means of monitoring learning progress without the need for explicit testing. However, applying stealth assessment is a complex and laborious process. This paper analyses the limitations of stealth assessment and conceptualizes the requirements for developing a generic tool that could overcome its barriers and accommodate its practical application. Hence, a framework is presented describing its user and functional requirements. The proposed generic solution could open up the wider uptake of stealth assessment in serious games.

Effectiveness of Serious Games (SG) depends very much on their capacity to provide the right balance be- tween gaming and educational experience. This require- ment raises challenges regarding realization of their in- telligence and personalization. We aim to overcome the problems of research fragmentation and identify some of the main issues by presenting a summary of relevant contributions from Artificial Intelligence and Personal- ization, together with a discussion on their future direc- tions. In this paper, we summarize approaches to user and learning goals modeling, user engagement, various levels of game adaptation, how new sensors and mobile technology can better identify the context of the user, content adaptation and reusability.

Designing an educational scenario is a sensitive and challenging activity because it is the vector of learning. However, the designed scenario may not correspond to some learners' characteristics (pace of work, cognitive styles, emotional factors, prerequisite knowledge, …). To personalize the learning task and adapt it gradually to each learner, several scenarios are needed. Adaptation and personalization are difficult because it is necessary on the one hand to know in advance the profiles and on the other hand to produce the multiple scenarios corresponding to these profiles. Our model allows to design many scenarios without knowing the learner profiles beforehand. Furthermore, it offers each learner opportunities to choose a scenario and to change it during their learning process. The model ensures that all announced objectives have enough resources for acquiring knowledge and activities for evaluation.

Project-based learning (PBL) enables learners to carry out challenging and authentic projects through investigations. However, PBL is difficult to implement successfully because learners often lack the selfregulation skills required to monitor, reflect, manage and assess their project activities and learning. Furthermore, most learning systems rarely offer possibilities to monitor and reflect on their project and learning processes. Hence, in this paper, we propose a general architecture of project-based learning management system. It enhances learners' reflective processes by supporting them in creating customisable indicators by exploring their traces. We developed an implementation named DDART and conducted an experiment in order to evaluate its usability and perceived utility. We found that this system supports learners to reflect and regulate their activities and learning, even if the indicator creation could be difficult for the novices.

Educational games are one prominent type of Serious Games that are designed for educational purposes. One goal of evaluating their benefits is ensuring that they meet these objectives. Some measures for this purpose are defined in fields like Learning Analytics and Technology-enhanced Learning. However, unlike other educational methods, meeting learning goals is not the only success measure of Serious Games. Contrary, complex requirements arise from their nature as games which are discussed in fields like Game User Research and Game Analytics and need to be considered. This thesis investigates how the use of new data collection opportunities enabled by smartphones as an interaction platform can benefit research in the field of Serious Games evaluation. Unobtrusive integrated sensors are used for observing user interactions in natural contexts. This can be vital for capturing factors that can help evaluate a Serious Game from multiple perspectives and with deeper insights. With the l...

Facial behavior analysis is a broad topic with various categories such as facial emotion recognition, age, and gender recognition. Many studies focus on individual tasks while the multi-task learning approach is still an open research issue and requires more research. In this paper, we present our solution and experiment result for the Multi-Task Learning challenge of the Affective Behavior Analysis in-the-wild competition. The challenge is a combination of three tasks: action unit detection, facial expression recognition, and valance-arousal estimation. To address this challenge, we introduce a cross-attentive module to improve multi-task learning performance. Additionally, a facial graph is applied to capture the association among action units. As a result, we achieve the evaluation measure of 128.8 on the validation data provided by the organizers, which outperforms the baseline result of 30.

The teaching of communication skills is a labour-intensive task because of the detailed feedback that should be given to learners during their prolonged practice. This study investigates to what extent our FILTWAM facial and vocal emotion recognition software can be used for improving a serious game (the Communication Advisor) that delivers a web-based training of communication skills. A test group of 25 participants played the game wherein they were requested to mimic specific facial and vocal emotions. Half of the assignments included direct feedback and the other half included no feedback. It was investigated whether feedback on the mimicked emotions would lead to better learning. The results suggest the facial performance growth was found to be positive, particularly significant in the feedback condition. The vocal performance growth was significant in both conditions. The results are a significant indication that the automated feedback from the software improves learners' communication performances.

This article describes the validation study of our software that uses combined webcam and microphone data for real-time, continuous, unobtrusive emotion recognition as part of our FILTWAM framework. FILTWAM aims at deploying a real-time multimodal emotion recognition method for providing more adequate feedback to the learners through an online communication skills training. Herein, timely feedback is needed that reflects on the intended emotions they show and which is also useful to increase learners' awareness of their own behavior. At least, a reliable and valid software interpretation of performed face and voice emotions is needed to warrant such adequate feedback. This validation study therefore calibrates our software. The study uses a multimodal fusion method. Twelve test persons performed computer-based tasks in which they were asked to mimic specific facial and vocal emotions. All test persons' behavior was recorded on video and two raters independently scored the showed emotions, which were contrasted with the software recognition outcomes. A hybrid method for multimodal fusion of our multimodal software shows accuracy between 96.1% and 98.6% for the best-chosen WEKA classifiers over predicted emotions. The software fulfils its requirements of real-time data interpretation and reliable results.

This paper presents the voice emotion recognition part of the FILTWAM framework for real-time emotion recognition in affective e-learning settings. FILT WAM (Framework for Improving Learning Through Webcams And Microphones) intends to offer timely and appropriate online feedback based upon learner's vocal intonations and facial expressions in order to foster their learning. Whereas the facial emotion recognition part has been successfully tested in a previous study, the here presented study describes the development and testing of FILTWAM's vocal emotion recognition software artefact. The main goal of this study was to show the valid use of computer microphone data for real-time and adequate interpretation of vocal intonations into extracted emotional states. The software that was developed was tested in a study with 12 participants. All participants individually received the same computerbased tasks in which they were requested 80 times to mimic specific vocal expressions (960 occurrences in total). Each individual session was recorded on video. For the validation of the voice emotion recognition software artefact, two experts annotated and rated participants' recorded behaviours. Expert findings were then compared with the software recognition results and showed an overall accuracy of Kappa of 0.743. The overall accuracy of the voice emotion recognition software artefact is 67 % based on the requested emotions and the recognized emotions. Our FILTWAM-software allows to continually and unobtrusively observing learners' behaviours and transforms these behaviours into emotional states. This paves the way for unobtrusive and real-time capturing of learners' emotional states for enhancing adaptive e-learning approaches.

This paper addresses the implications of combining learning analytics and serious games for improving game quality, monitoring and assessment of player behavior, gaming performance, game progression, learning goals achievement, and user's appreciation. We introduce two modes of serious games analytics: in-game (real time) analytics, and post-game (off-line) analytics. We also explain the GLEANER framework for in-game analytics and describe a practical example for offline analytics. We conclude with a brief outlook on future work, highlighting opportunities and challenges towards a solid uptake of SGs in authentic educational and training settings.

This paper presents a framework (FILTWAM) for real time emotion recognition in e-learning by using webcams. FILTWAM (Framework for Improving Learning Through Webcams And Microphones) offers timely and relevant feedback based upon learner's facial expressions and verbalizations. FILTWAM's facial expression software module has been developed and tested in a proof of concept study. The main goal of this study was to validate the use of webcam data for a real-time and adequate interpretation of facial expressions into extracted emotional states. The software was calibrated with ten test persons. They received the same computer-based tasks in which each of them were requested a hundred times to mimic specific facial expressions. All sessions were recorded on video. For the validation of the face emotion recognition software, two experts annotated and rated participants' recorded behaviours. Expert findings were contrasted with the software results and showed an overall value of Kappa of 0.77. An overall accuracy of our software based on the requested emotions and the recognized emotions is 72%. Whereas existing software only allows not-real time, discontinuous and obtrusive facial detection, our software allows to continuously and unobtrusively monitor learners' behaviours and converts these behaviours directly into emotional states. This paves the way for enhancing the quality and efficacy of e-learning by including the learner's emotional states.

This paper introduces the voice emotion recognition part of our framework for improving learning through webcams and microphones (FILTWAM). This framework enables multimodal emotion recognition of learners during game-based learning. The main goal of this study is to validate the use of microphone data for a real-time and adequate interpretation of vocal expressions into emotional states were the software is calibrated with end users. FILTWAM already incorporates a valid face emotion recognition module and is extended with a voice emotion recognition module. This extension aims to provide relevant and timely feedback based upon learner's vocal intonations. The feedback is expected to enhance learner's awareness of his or her own behavior. Six test persons received the same computer-based tasks in which they were requested to mimic specific vocal expressions. Each test person mimicked 82 emotions, which led to a dataset of 492 emotions. All sessions were recorded on video. An overall accuracy of our software based on the requested emotions and the recognized emotions is a pretty good 74.6% for the emotions happy and neutral emotions; but will be improved for the lower values of an extended set of emotions. In contrast with existing software our solution allows to continuously and unobtrusively monitor learners' intonations and convert these intonations into emotional states. This paves the way for enhancing the quality and efficacy of game-based learning by including the learner's emotional states, and links these to pedagogical scaffolding.

This paper introduces a Framework for Improving Learning Through Webcams And Microphones (FILTWAM). It proposes an overarching framework comprising conceptual and technical frameworks for enhancing the online communication skills of lifelong learners. Our approach interprets the emotional state of people using webcams and microphones and combines relevant and timely feedback based upon learner's facial expressions and verbalizations (like sadness, anger, disgust, fear, happiness, surprise, and neutral). The feedback generated from the webcams is expected to to-face trainings and better helps the interests of lifelong learners who prefer to study at their own pace, place and time. Our small-scale proof of concept study exemplifies the practical application of FILTWAM and provides first evaluation results on that. This study will guide future development of software, training materials, and research. It will validate the use of webcam data for a real-time and adequate interpretation of facial expressions into emotional states. Participants' behaviour is recorded on videos so that videos will be replayed, rated, annotated and evaluated by expert observers and contrasted with participants' own opinions in future research.

This paper addresses the implications of combining learning analytics and serious games for improving game quality, monitoring and assessment of player behavior, gaming performance, game progression, learning goals achievement, and user's appreciation. We introduce two modes of serious games analytics: in-game (real time) analytics, and post-game (off-line) analytics. We also explain the GLEANER framework for in-game analytics and describe a practical example for offline analytics. We conclude with a brief outlook on future work, highlighting opportunities and challenges towards a solid uptake of SGs in authentic educational and training settings.

In recent years, the main problem in e-learning has shifted from analyzing content to personalization of learning environment by Intelligence Tutoring Systems (ITSs). Therefore, by designing personalized teaching models, learners are able to have a successful and satisfying experience in achieving their learning goals. Affective Tutoring Systems (ATSs) are some kinds of ITS that can recognize and respond to affective states of learners. In this study, we designed, implemented, and evaluated a system to personalize the learning environment based on the facial emotions recognition, head pose estimation, and cognitive style of learners. First, a unit called Intelligent Analyzer (AI) created which was responsible for recognizing facial expression and head angles of learners. Next, the ATS was built which mainly made of two units: ITS, IA. Results indicated that with the ATS, participants needed less efforts to pass the tests. In other words, we observed when the IA unit was activated, learners could pass the final tests in fewer attempts than those for whom the IA unit was deactivated. Additionally, they showed an improvement in terms of the mean passing score and academic satisfaction.

Automatic affect recognition in real-world environments is an important task towards a natural interaction between humans and machines. The recent years, several advancements have been accomplished in determining the emotional states with the use of Deep Neural Networks (DNNs). In this paper, we propose an emotion recognition system that utilizes the raw text, audio and visual information in an end-to-end manner. To capture the emotional states of a person, robust features need to be extracted from the various modalities. To this end, we utilize Convolutional Neural Networks (CNNs) and propose a novel transformer-based architecture for the text modality that can robustly capture the semantics of sentences. We develop an audio model to process the audio channel, and adopt a variation of a high resolution network (HRNet) to process the visual modality. To fuse the modality-specific features, we propose novel attention-based methods. To capture the temporal dynamics in the signal, we utilize Long Short-Term Memory (LSTM) networks. Our model is trained on the SEWA dataset of the AVEC 2017 research sub-challenge on emotion recognition, and produces state-of-the-art results in the text, visual and multimodal domains, and comparable performance in the audio case when compared with the winning papers of the challenge that use several hand-crafted and DNN features. Code is available at: https://github.com/glam-imperial/multimodal-affect-recognition.

In recent years, the main problem in e-learning has shifted from analyzing content to personalization of learning environment by Intelligence Tutoring Systems (ITSs). Therefore, by designing personalized teaching models, learners are able to have a successful and satisfying experience in achieving their learning goals. Affective Tutoring Systems (ATSs) are some kinds of ITS that can recognize and respond to affective states of learners. In this study, we designed, implemented, and evaluated a system to personalize the learning environment based on the facial emotions recognition, head pose estimation, and cognitive style of learners. First, a unit called Intelligent Analyzer (AI) created which was responsible for recognizing facial expression and head angles of learners. Next, the ATS was built which mainly made of two units: ITS, IA. Results indicated that with the ATS, participants needed less efforts to pass the tests. In other words, we observed when the IA unit was activated, l...

Learning analytics primarily deals with the optimization of learning environments and the ultimate goal of learning analytics is to improve learning and teaching efficiency. Studies on learning analytics seem to have been made in the form of adaptation engine and intervention engine. Adaptation engine studies are quite widespread, but intervention engine work has been seen to studies very few. For the intervention engine studies, it was generally determined that interventions were made with feedback and dashboards. The aim of this study is to reveal an intervention engine framework which is based on learning analytics. Within this framework, a system design has been put forward which can provide instructional, supportive and motivational interventions to learners. These interventions are based on both the learning outputs of the learners and their learning experiences. Instructional interventions are planned to be based on learning outputs, supportive and motivational interventions ...

Learning analytic dashboards (LADs) are data visualization systems that use dynamic data in digital learning environments to provide students, teachers, and administrators with a wealth of information about student’s engagement, experiences, and performance on tasks. LADs have become increasingly popular, particularly in formative learning contexts, and help teachers make data-informed decisions about a student’s developing skills on a topic. LADs afford the possibility for teachers to obtain real-time data on student performance, response processes, and progress on academic learning tasks. However, data presented on LADs are often not based on an evaluation of stakeholder needs, and have been found to not be clearly interpretable and actionable for teachers to readily adapt their pedagogical actions based on these insights. We elaborate on how insights from research focused on interpretation and use of Score Reporting systems and research on open learner models (OLMs) can be used t...