FILTWAM and Voice Emotion Recognition

Towards Multimodal Emotion Recognition in E-learning Environments, 2014

This paper presents a framework (FILTWAM (Framework for Improving Learning Through Webcams And Microphones)) for real-time emotion recognition in e-learning by using webcams. FILTWAM offers timely and relevant feedback based upon learner's facial expressions and verbalizations. FILTWAM's facial expression software module has been developed and tested in a proof-of-concept study. The main goal of this study was to validate the use of webcam data for a real-time and adequate interpretation of facial expressions into extracted emotional states. The software was calibrated with 10 test persons. They received the same computer-based tasks in which each of them were requested 100 times to mimic specific facial expressions. All sessions were recorded on video. For the validation of the face emotion recognition software, two experts annotated and rated participants’ recorded behaviours. Expert findings were contrasted with the software results and showed an overall value of kappa of 0.77. An overall accuracy of our software based on the requested emotions and the recognized emotions is 72%. Whereas existing software only allows not-real time, discontinuous and obtrusive facial detection, our software allows to continuously and unobtrusively monitor learners' behaviours and converts these behaviours directly into emotional states. This paves the way for enhancing the quality and efficacy of e-learning by including the learner's emotional states.

Cognition and …, 2008

In an attempt to discover the facial action units for affective states that occur during complex learning, this study adopted an emote-aloud procedure in which participants were recorded as they verbalised their affective states while interacting with an intelligent tutoring system (AutoTutor). Participants' facial expressions were coded by two expert raters using Ekman's Facial Action Coding System and analysed using association rule mining techniques. The two expert raters received an overall kappa that ranged between .76 and .84. The association rule mining analysis uncovered facial actions associated with confusion, frustration, and boredom. We discuss these rules and the prospects of enhancing AutoTutor with non-intrusive affect-sensitive capabilities.

Emotion, 2005

To determine the degree to which emotional changes in speech reflect factors other than arousal, such as valence, the authors used a computer game to induce natural emotional speech. Voice samples were elicited following game events that were either conducive or obstructive to the goal of winning and were accompanied by either pleasant or unpleasant sounds. Acoustic analysis of the speech recordings of 30 adolescents revealed that mean energy, fundamental-frequency level, utterance duration, and the proportion of an utterance that was voiced varied with goal conduciveness; spectral energy distribution depended on manipulations of pleasantness; and pitch dynamics depended on the interaction of pleasantness and goal conduciveness. The results suggest that a single arousal dimension does not adequately characterize a number of emotion-related vocal changes, lending weight to multidimensional theories of emotional response patterning.

Communication & cognition. Monographies, 2010

Emotions are assumed to have a great impact on our behaviour and also on our learning behaviour. In a face-to-face learning environment emotions can be expressed by (non-)verbal behaviour as way of speaking, facial expressions or words with emotional loadings. In this paper we research the possibility to assess the emotional state of learners by analysis of nonverbal behaviour as speech analysis and analysis of facial expressions. We developed tools to extract features from sound and video recordings and used classifiers as SVM to label emotional states. We used discrete emotional states but also the well-known 2D valence and arousal score as a continuous score of the emotional state. From our experiments it proves that students show overt emotions under special conditions with strong emotional triggers. Our system was able to assess strong emotions up to some level.

In D. Ifenthaler, D. Eseryel & X. Ge (Eds.), Assessment in game-based learning: Foundations, innovations, and perspectives. New York Springer , 2012

Research has shown that emotions are directly linked to cognition and there is a strong correlation between affect and learning. This notion along with recent technological advancements has prompted researchers from many disciplines to turn their attention toward adding an affective component to humancomputer dialog. This chapter discusses emotion assessment methods, recent empirical research related to examining students’ affective states in entertainment and educational games, and conceptual, methodological, and technological issues associated with developing emotion recognition models. An overview of emotion recognition research suggests that there is little consensus on what emotions should be measured and how to do it. Moreover, it is still not clear how emotions affect human learning and performance.

2010 International Conference on Cyberworlds, 2010

Collaborative Virtual Environments (CVEs) have become increasingly popular in the past two decades. Most CVEs use avatar systems to represent each user logged into a CVE session. Some avatar systems are capable of expressing emotions with postures, gestures, and facial expressions. In previous studies, various approaches have been explored to convey emotional states to the computer, including voice and facial movements. We propose a technique to detect emotions in the voice of a speaker and animate avatars to reflect extracted emotions in real-time. The system has been developed in "Project Wonderland," a Java-based open-source framework for creating collaborative 3D virtual worlds. In our prototype, six primitive emotional states-anger, dislike, fear, happiness, sadness, and surprise-were considered. An emotion classification system which uses short time log frequency power coefficients (LFPC) to represent features and hidden Markov models (HMMs) as the classifier was modified to build an emotion classification unit. Extracted emotions were used to activate existing avatar postures and gestures in Wonderland.

EURAS Journal of Engineering and Applied Sciences

During the Covid-19 pandemic, everything has been shifted to using the internet to get whatever the job is done. In particular, the educational system used the internet for online classes and meetings. This virtual connection faced some issues such as no facial expression between the students and the teachers. This is a big issue when it comes to learning. Neither the teacher nor the student has got to understand everything in the online classroom. This study comes to a solution. So, in order to make those virtual classrooms as normal as the face-to-face classroom, it creates a system of capturing the Emotion expressions into understandable language. This language is data that transforms into words. So, the teacher could get a statement about the student's emotions. Emotion recognition (ER) is the process of identifying human emotion, and it's the most complex field of pattern recognition. It has become an even more important topic since it has become the main component in many Artificial Intelligence applications. The goal of emotion recognition is to identify human feelings. Feelings can be captured either from the face or from verbal communication. Emotion recognition is a valuable tool for a variety of industries and applications. It can improve security measures, aid in healthcare, and assist in the care of children with autism. Additionally, it is utilized in the development of video games and human-computer interactions. One example of its use is in a study that uses students' computers to capture facial expressions through a camera, helping to identify common emotions.

International Journal of Computer and Electrical Engineering, 2012

Constructive learning is an important research area having wide impact on teaching methods in education, learning theories, and plays a major role in many education reform movements. Teachers play a major role in improving the learning skill of the students. It is observed that constructive learning advocates the interconnection between emotions and learning. When students fail to get the excepted results they tend to feel that they are not good at the subject/task. Teachers should make them realize that failure is also a part of learning process and improve their learning rates. Human teachers identify the emotions of students with varying degrees of accuracy. In learning with computers, computers also should be given the capability to recognize emotions so as to optimize the learning process. Literature survey indicates the wide use of image processing to understand the constructive learning theory. The paper presents a novel system which can be used by computer to access the emotional state of the learner further presenting a corrective measure to improve their learning states. This is the first paper which analyses constructive learning using speech analysis. It is the primary paper which analyses the effect of emotions on the learning rate using pitch tracking. A database consisting of acoustic waveforms produced by an amateur musician is taken and the learning rates are analyzed. The pitch contours of waveforms are compared with the standard waveform and the error graphs are plotted. Analysis of the emotion of the subject is also made by observing the error plots.

Lrec, 2010

In this paper we describe a corpus set together from two sub-corpora. The CINEMO corpus contains acted emotional expression obtained by playing dubbing exercises. This new protocol is a way to collect mood-induced data in large amount which show several complex and shaded emotions. JEMO is a corpus collected with an emotion-detection game and contains more prototypical emotions than CINEMO. We show how the two sub-corpora balance and enrich each other and result in a better performance. We built male and female emotion models and use Sequential Fast Forward Feature Selection to improve detection performances. After feature-selection we obtain good results even with our strict speaker independent testing method. The global corpus contains 88 speakers (38 females, 50 males). This study has been done within the scope of the ANR (National Research Agency) Affective Avatar project which deals with building a system of emotions detection for monitoring an Artificial Agent by voice.

While significant work is being done in order to develop empathic agents which can identify emotions of the user through eye gaze and facial expressions, a neglected area, especially in the pedagogical context, is the use of voice for detection of alertness, fatigue and emotions. Some of the issues are lack of constant monitoring and visual feedback. However, such a system has advantages -the ability to work where visual monitoring is expensive, in darkness or where mobile devices cannot provide adequate visual feedback. We propose a model for a system capable of identifying emotions as well as alertness and fatigue based entirely on voice interaction, keyboard and mouse clicks; we also propose to develop an engine which can intelligently improve its prediction of emotions and cognitive states based on earlier interaction, and suggest appropriate measures to improve emotions, reduce distractions and mitigate fatigue.