Bimodal Emotion Recognition using Speech and Physiological Changes

Internet Imaging VI, 2005

Recent technological advances have enabled human users to interact with computers in ways previously unimaginable. Beyond the confines of the keyboard and mouse, new modalities for human-computer interaction such as voice, gesture, and force-feedback are emerging. Despite important advances, one necessary ingredient for natural interaction is still missing-emotions. Emotions play an important role in human-to-human communication and interaction, allowing people to express themselves beyond the verbal domain. The ability to understand human emotions is desirable for the computer in several applications. This paper explores new ways of human-computer interaction that enable the computer to be more aware of the user's emotional and attentional expressions. We present the basic research in the field and the recent advances into the emotion recognition from facial, voice, and pshysiological signals, where the different modalities are treated independently. We then describe the challenging problem of multimodal emotion recognition and we advocate the use of probabilistic graphical models when fusing the different modalities. We also discuss the difficult issues of obtaining reliable affective data, obtaining ground truth for emotion recognition, and the use of unlabeled data.

Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition, 1998

Recognizing human facial expression and emotion by computer is an interesting and challenging problem. Many have investigated emotional contents in speech alone, or recognition of human facial expressions solely from images. However, relatively little has been done in combining these two modalities for recognizing human emotions. De Silva et al. 4 studied human subjects' ability to recognize emotions from viewing video clips of facial expressions and listening to the corresponding emotional speech stimuli. They found that humans recognize some emotions better by audio information, and other emotions better by video. They also proposed an algorithm to integrate both kinds of inputs to mimic human's recognition process. While attempting to implement the algorithm, we encountered di culties which led us to a di erent approach. We found these two modalities to be c omplimentary. By using both, we show it is possible to achieve higher recognition rates than either modality alone.

1998

In this paper, we report preliminary results of recognizing emotions using both speech and video by machine. We applied automatic feature analysis and extraction algorithms to the same data used by De Silva et al. 5] in their human subjective studies. We compare machine recognition results to human performances in three tests:(1) video only, (2) audio only, (3) combined audio and video. In machine-assisted analyses, we found these two modalities to be complementary. Emotion categories that have similar features in one modality have very di erent features in the other modality. By using both, we show it is possible to achieve higher recognition rates than either modality alone.

Proceedings of the Second …, 2010

When interacting with robots we show a plethora of affective reactions typical of natural communications. Indeed, emotions are embedded on our communications and represent a predominant communication channel to convey relevant, high impact, information. In recent years more and more researchers have tried to exploit this channel for human robot (HRI) and human computer interactions (HCI). Two key abilities are needed for this purpose: the ability to display emotions and the ability to automatically recognize them. In this work we present our system for the computer based automatic recognition of emotions and the new results we obtained on a small dataset of quasi unconstrained emotional videos extracted from TV series and movies. The results are encouraging showing a recognition rate of about 74%.

2008 IEEE International Symposium on Signal Processing and Information Technology, 2008

Affective and human-centered computing have attracted a lot of attention during the past years, mainly due to the abundance of environments and applications able to exploit and adapt to multimodal input from the users. The combination of facial expressions with prosody information allows us to capture the users' emotional state in an unintrusive manner, relying on the best performing modality in cases where one modality suffers from noise or bad sensing conditions. In this paper, we describe a multi-cue, dynamic approach to detect emotion in naturalistic video sequences, where input is taken from nearly real world situations, contrary to controlled recording conditions of audiovisual material. Recognition is performed via a recurrent neural network, whose short term memory and approximation capabilities cater for modeling dynamic events in facial and prosodic expressivity. This approach also differs from existing work in that it models user expressivity using a dimensional representation, instead of detecting discrete 'universal emotions', which are scarce in everyday human-machine interaction. The algorithm is deployed on an audiovisual database which was recorded simulating human-human discourse and, therefore, contains less extreme expressivity and subtle variations of a number of emotion labels. Results show that in turns lasting more than a few frames, recognition rates rise to 98%. moving away from the monolithic window-mouse-pointer interface paradigm and utilizing more intuitive concepts, closer to human niches ([47], [48]). A large part of this naturalistic interaction concept is expressivity [49], both in terms of interpreting the reaction of the user to a particular event or taking into account their emotional state and adapting presentation to it, since it alleviates the learning curve for conventional interfaces and makes less technology-savvy users feel more comfortable. In this framework, both speech and facial expressions are of great importance, since they usually provide a comprehensible view of users' reactions; actually, Cohen commented on the emergence and significance of multimodality, albeit in a slightly different human-computer interaction (HCI) domain, in [55] and [56], while Oviatt [50] indicated that an interaction pattern constrained to mere 'speak-and-point' only makes up for a very small fraction of all spontaneous multimodal utterances in everyday HCI [51]. In the context of HCI, [54] defines a multimodal system as one that 'responds to inputs in more than one modality or communication channel' abundance, while Mehrabian [52] suggests that facial expressions and vocal intonations are the main means for someone to estimate a person's affective state [53], with the face being more accurately judged, or correlating better with judgments based on full audiovisual input than on voice input ([54], [57]). This fact led to a number of approaches using video and audio to tackle emotion recognition in a multimodal manner ([3], [58]-[63], [67]), while recently the visual modality has been extended to include facial, head or body gesturing ([64] and [65], extended in [66]). Additional factors that contribute to the complexity of estimating expressivity in everyday HCI are the fusion of the information extracted from modalities ([50]), the interpretation of the data through time and the noise and uncertainty alleviation from the natural setting ([4], [71]). In the case of fusing multimodal information [72], systems can either integrate signals at the feature level ([73]) or, after coming up with a class decision at the feature level of each modality, by merging decisions at a semantic level (late identification, [73] and [74]), possibly taking into account any confidence measures provided by each modality or, generally, a mixture of experts mechanism [6]. Regarding the dynamic nature of expressivity, Littlewort [76] states that while muscle-based techniques can describe the morphology of a facial expression, it is very difficult for them to illustrate in a measurable (and, therefore detectable) manner the dynamics, i.e. the temporal pattern of muscle activation and observable feature movement or deformation. She also makes a case of natural expressivity being inherently different in temporal terms than posed, presenting arguments from psychologists ([77] and [78]), proving the dissimilarity of posed and natural data, in addition to the need to tackle expressivity using mechanisms that capture dynamic attributes. As a general rule, the naturalistic data chosen as input in this work, is closer to human reality since intercourse is not acted and expressivity is not guided by directives (e.g. Neutral expression à one of the six universal emotions à neutral). This amplifies the difficulty in discerning facial expressions and speech patterns [70]. Nevertheless it provides the perfect test-bed for the combination of the conclusions drawn from each modality in one time unit and use as input in the following sequence of audio and visual events analyzed.

ArXiv, 2020

The primary objective is to teach a machine about human emotions, which has become an essential requirement in the field of social intelligence, also expedites the progress of human-machine interactions. The ability of a machine to understand human emotion and act accordingly has been a choice of great interest in today's world. The future generations of computers thus must be able to interact with a human being just like another. For example, people who have Autism often find it difficult to talk to someone about their state of mind. This model explicitly targets the userbase who are troubled and fail to express it. Also, this model's speech processing techniques provide an estimate of the emotion in the case of poor video quality and vice-versa.

Human affective behavior is multimodal, continuous and complex. Despite major advances within the affective computing research field, modeling, analyzing, interpreting and responding to human affective behavior still remains a challenge for automated systems. Therefore, affective and behavioral computing researchers have recently invested increased effort in exploring how to best model, analyze and interpret the subtlety, complexity and continuity of affective behavior in terms of latent dimensions (e.g., arousal, power and valence) and appraisals, rather than in terms of a small number of discrete emotion categories (e.g., happiness and sadness). This chapter aims to (i) give a brief overview of the existing efforts and the major accomplishments in modeling and analysis of emotional expressions in dimensional and continuous space while focusing on open issues and new challenges in the field, and (ii) introduce a representative approach for multimodal continuous analysis of affect from voice and face.

SHS Web of Conferences, 2021

As emotional content reflects human behaviour, automatic emotion recognition is a topic of growing interest. During the communication of an emotional message, the use of physiological signals and facial expressions gives several advantages that can be expected to understand a person’s personality and psychopathology better and determine human communication and human-machine interaction. In this article, we will present some notions about identifying the emotional state through visual expression, auditory expression and physiological representation, and the techniques used to measure emotions.

Affect and emotion in human- …, 2008

2010

Recent advances in human-computer interaction technology go beyond the successful transfer of data between human and machine by seeking to improve the naturalness and friendliness of user interactions. An important augmentation, and potential source of feedback, comes from recognizing the user's expressed emotion or affect. This chapter presents an overview of research efforts to classify emotion using different modalities: audio, visual and audiovisual combined. Theories of emotion provide a framework for defining emotional categories or classes. The first step, then, in the study of human affect recognition involves the construction of suitable databases. The authors describe fifteen audio, visual and audiovisual data sets, and the types of feature that researchers have used to represent the emotional content. They discuss data-driven methods of feature selection and reduction, which discard noise and irrelevant information to maximize the concentration of useful information. They focus on the popular types of classifier that are used to decide to which emotion class a given example belongs, and methods of fusing information from multiple modalities. Finally, the authors point to some interesting areas for future investigation in this field, and conclude. Recent work in the field of emotion recognition involves combining the audio and visual modalities to improve the performance of emotion recognition systems. This has resulted in the recording of audio