Multimodal Emotion Recognition and Expressivity Analysis

2012 Eighth International Conference on Signal Image Technology and Internet Based Systems, 2012

Emotions play a crucial role in human-computer interaction. They are generally expressed and perceived through multiple modalities such as speech, facial expressions, physiological signals. Indeed, the complexity of emotions makes the acquisition very difficult and makes unimodal systems (i.e., the observation of only one source of emotion) unreliable and often unfeasible in applications of high complexity. Moreover the lack of a standard in human emotions modeling hinders the sharing of affective information between applications. In this paper, we present a multimodal approach for the emotion recognition from many sources of information. This paper aims to provide a multi-modal system for emotion recognition and exchange that will facilitate inter-systems exchanges and improve the credibility of emotional interaction between users and computers. We elaborate a multimodal emotion recognition method from Physiological Data based on signal processing algorithms. Our method permits to recognize emotion composed of several aspects like simulated and masked emotions. This method uses a new multidimensional model to represent emotional states based on an algebraic representation. The experimental results show that the proposed multimodal emotion recognition method improves the recognition rates in comparison to the unimodal approach. Compared to the state of art multimodal techniques, the proposed method gives a good results with 72% of correct.

Proceedings of the ACM International …, 2010

Multimedia indexing is about developing techniques allowing people to effectively find media. Content-based methods become necessary when dealing with large databases as people cannot possibly annotate all available content. Emotions are intrinsic in human beings and are known to be very important for natural interactions, decision making, memory, and many other cognitive functions. Current technologies allows exploring the emotional space by mean of contentbased analysis of audio and video, but also thanks to other modalities such as the human physiology.

Journal on Multimodal …, 2010

IEEE Transactions on Image Processing, 2000

2015

HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et a ̀ la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Emotion detection of users is a challenging and exciting field where user's data is analysed to recognise emotions such as happy, sad, angry etc. This data could be in one or multiple formats such as audio, video ,text ,still images et al. Relevant features are extracted and fused together to give a label. Fusing data from two or more sources(modalities) is another challenge, feature level or decision level fusion is employed. This paper inspects and studies the various approaches to multimodal extraction of emotions.

Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition, 1998

Recognizing human facial expression and emotion by computer is an interesting and challenging problem. Many have investigated emotional contents in speech alone, or recognition of human facial expressions solely from images. However, relatively little has been done in combining these two modalities for recognizing human emotions. De Silva et al. 4 studied human subjects' ability to recognize emotions from viewing video clips of facial expressions and listening to the corresponding emotional speech stimuli. They found that humans recognize some emotions better by audio information, and other emotions better by video. They also proposed an algorithm to integrate both kinds of inputs to mimic human's recognition process. While attempting to implement the algorithm, we encountered di culties which led us to a di erent approach. We found these two modalities to be c omplimentary. By using both, we show it is possible to achieve higher recognition rates than either modality alone.

Negative emotions (anxiety, fear, anger, and grief) may affect physical health and the quality of life. Indeed, people with depression experience severe and prolonged feelings of negative emotions like sadness , anger, disgust and fear. On one hand, this paper presents a new method for the fusion of signals for the purpose of a multimodal recognition of eight basic emotions, on the other hand, it present a classification of these basic emotions in three emotional classes, namely, neutral, positive and negative emotions which are using physiological signals. After constructing an emotion data base during the learning phase, we apply the recognition algorithm on each modality separately. Then, we merge all these decisions separately by applying a decision fusion approach to improve recognition rate. The experiments show that the proposed method allows high accuracy emotion recognition. Indeed, we get a recognition rate of 81.69% under some conditions.

2005

New theories on emotion emphasize the tight boundaries among emotion, motivation and cognition: according to them people are motivated to respond to events differently depending on how they have been appraised. The present study tries to examine what kind of emotional responses the subject is motivated to express while interacting with an artificial agent if he/she believes that this agent is able to understand his/her emotional states. For this purpose different kind of computer games were projected to elicit specific emotional appraisals and two different conditions were used: in one condition the avatar provided a simulated intelligent feedback, while in the second only guided the subject across the different tasks. Multimodal synchronized data were captured. All video tapes were codified frame by frame using The Observer 5.0 and THEME Software. This is an in-progress study and data are still under elaboration. This paper includes initial results of the analysis of non-verbal communicative signals.

International Journal of Applied Evolutionary Computation, 2000

Emotions play a crucial role in human-computer interaction. They are generally expressed and perceived through multiple modalities such as speech, facial expressions, physiological signals. Indeed, the complexity of emotions makes the acquisition very difficult and makes unimodal systems (i.e., the observation of only one source of emotion) unreliable and often unfeasible in applications of high complexity. Moreover the lack of a standard in human emotions modeling hinders the sharing of affective information between applications. In this paper, we present a multimodal approach for the emotion recognition from many sources of information. This paper aims to provide a multi-modal system for emotion recognition and exchange that will facilitate inter-systems exchanges and improve the credibility of emotional interaction between users and computers. We elaborate a multimodal emotion recognition method from Physiological Data based on signal processing algorithms. Our method permits to recognize emotion composed of several aspects like simulated and masked emotions. This method uses a new multidimensional model to represent emotional states based on an algebraic representation. The experimental results show that the proposed multimodal emotion recognition method improves the recognition rates in comparison to the unimodal approach. Compared to the state of art multimodal techniques, the proposed method gives a good results with 72% of correct.