Multimodal human emotion/expression recognition

International Journal of Advanced Computer Science and Applications, 2022

This Emotion recognition plays a prominent role in today's intelligent system applications. Human computer interface, health care, law, and entertainment are a few of the applications where emotion recognition is used. Humans convey their emotions in the form of text, voice, and facial expressions, thus developing a multimodal emotional recognition system playing a crucial role in human-computer or intelligent system communication. The majority of established emotional recognition algorithms only identify emotions in unique data, such as text, audio, or image data. A multimodal system uses information from a variety of sources and fuses the information by using fusion techniques and categories to improve recognition accuracy. In this paper, a multimodal system to recognise emotions was presented that fuses the features from information obtained from heterogenous modalities like audio and video. For audio feature extraction energy, zero crossing rate and Mel-Frequency Cepstral Coefficients (MFCC) techniques are considered. Of these, MFCC produced promising results. For video feature extraction, first the videos are converted to frames and stored in a linear scale space by using a spatial temporal Gaussian Kernel. The features from the images are further extracted by applying a Gaussian weighted function to the second momentum matrix of linear scale space data. The Marginal Fisher Analysis (MFA) fusion method is used to fuse both the audio and video features, and the resulted features are given to the FERCNN model for evaluation. For experimentation, the RAVDESS and CREMAD datasets, which contain audio and video data, are used. Accuracy levels of 95.56, 96.28, and 95.07 on the RAVDESS dataset and accuracies of 80.50, 97.88, and 69.66 on the CREMAD dataset in audio, video, and multimodal modalities are achieved, whose performance is better than the existing multimodal systems.

Negative emotions (anxiety, fear, anger, and grief) may affect physical health and the quality of life. Indeed, people with depression experience severe and prolonged feelings of negative emotions like sadness , anger, disgust and fear. On one hand, this paper presents a new method for the fusion of signals for the purpose of a multimodal recognition of eight basic emotions, on the other hand, it present a classification of these basic emotions in three emotional classes, namely, neutral, positive and negative emotions which are using physiological signals. After constructing an emotion data base during the learning phase, we apply the recognition algorithm on each modality separately. Then, we merge all these decisions separately by applying a decision fusion approach to improve recognition rate. The experiments show that the proposed method allows high accuracy emotion recognition. Indeed, we get a recognition rate of 81.69% under some conditions.

Proceedings of the 2020 International Conference on Multimodal Interaction, 2020

Proceedings of the ACM International …, 2010

Multimedia indexing is about developing techniques allowing people to effectively find media. Content-based methods become necessary when dealing with large databases as people cannot possibly annotate all available content. Emotions are intrinsic in human beings and are known to be very important for natural interactions, decision making, memory, and many other cognitive functions. Current technologies allows exploring the emotional space by mean of contentbased analysis of audio and video, but also thanks to other modalities such as the human physiology.

Multimodal Technologies and Interaction

Emotion classification is a research area in which there has been very intensive literature production concerning natural language processing, multimedia data, semantic knowledge discovery, social network mining, and text and multimedia data mining. This paper addresses the issue of emotion classification and proposes a method for classifying the emotions expressed in multimodal data extracted from videos. The proposed method models multimodal data as a sequence of features extracted from facial expressions, speech, gestures, and text, using a linguistic approach. Each sequence of multimodal data is correctly associated with the emotion by a method that models each emotion using a hidden Markov model. The trained model is evaluated on samples of multimodal sentences associated with seven basic emotions. The experimental results demonstrate a good classification rate for emotions.

Communications on Applied Nonlinear Analysis ISSN: 1074-133X, 2024

With improvements in multimodal data integration, which uses voice, text, and visual cues to correctly guess emotional states, emotion recognition has come a long way. In this study, we examine how adding facial expressions to speech and text sources can make emotion recognition systems more accurate and detailed. Including facial signals is vital since it includes a visual component that makes a difference us get it feelings more profoundly, complementing what able to learn from tuning in and perusing. In this think about, we utilize profound learning methods, particularly Convolutional Neural Systems (CNNs) and Repetitive Neural Systems (RNNs), to way better handle and combine multi-dimensional information. A. CNNs are exceptionally great at sifting spatial features from facial expressions. B. Identifying little changes within the way facial muscles move that show diverse feelings. At the same time, RNNs handle grouping information from composed sources, capturing semantic setting and etymological nuances that offer assistance individuals express their sentiments. In this strategy, the multimodal information is preprocessed to coordinate spatial highlights and synchronize transient angles, guaranteeing information consistency over all modalities. Feature extraction procedures are utilized to extricate important designs from each media. Usually taken after by a combination handle that works in couple to create the data more valuable. The objective of this combination prepare is to evacuate superfluous information and make the assumption classification show more versatile to the clamor and variety that comes with real-world information. Measures utilized for assessment incorporate precision, exactness, review and F1 score. These are compared against well-known assumption datasets such as AffectNet and IEMOCAP. This ponder explores how well multimodal integration works. We conclude that compared with unimodal strategies, multimodal integration is more precise and solid in capturing complex enthusiastic reactions, and can be utilized in numerous real-world spaces, such as healthcare, human-computer interaction (MCI), and emotional computing. Accurate emotion tracking helps doctors diagnose mental health problems and keep an eye on patients, which makes healing treatments based on emotional states more effective.

ATR workshop on virtual …, 1998

In this paper, we report preliminary results of rec-ognizing emotions using both speech and video by ma-chine. We applied automatic feature analysis and ex-traction algorithms to the same data used by De Silva et al. 5] in their human subjective studies. We compare machine ...

Proceedings of the Second …, 2010

When interacting with robots we show a plethora of affective reactions typical of natural communications. Indeed, emotions are embedded on our communications and represent a predominant communication channel to convey relevant, high impact, information. In recent years more and more researchers have tried to exploit this channel for human robot (HRI) and human computer interactions (HCI). Two key abilities are needed for this purpose: the ability to display emotions and the ability to automatically recognize them. In this work we present our system for the computer based automatic recognition of emotions and the new results we obtained on a small dataset of quasi unconstrained emotional videos extracted from TV series and movies. The results are encouraging showing a recognition rate of about 74%.

Journal of Intelligent & Fuzzy Systems, 2023

Emotional AI is the next era of AI to play a major role in various fields such as entertainment, health care, and self-paced online education. Multimodal affect analysis refers to the techniques employed to identify and measure the "experience of emotion" expressed in content containing text, audio, images, and video. We propose a multimodal emotion recognition system exploiting different modalities: speech, motion capture, and text data. We start with multiple baseline unimodal emotion recognition systems and systematically enhance each of them: the component for emotion extraction from speech using a time-distributed approach, motion capture using a CNN-LSTM model, and text using a BERT-based model. The fusion significantly improves the accuracy of detecting emotions on the IEMOCAP dataset. Specifically, the enhanced speech and mocap model achieves emotion recognition accuracies of 65.08% and 67.51%, respectively, and the BERT-based text model achieves an accuracy of 72.60%. The fusion model, which combines the text, audio, and motion capture models, achieves 80.20% accuracy on IEMOCAP, significantly improving upon the state-of-the-art methods. Our results also show that our proposed model achieves 63.52% and 61.65% in 5-class and 7-class classification on the MELD dataset.

Cybernetics and Intelligent …, 2010

The ability to recognize emotions in natural human communications is known to be very important for mankind. In recent years, a considerable number of researchers have investigated techniques allowing computer to replicate this capability by analyzing both prosodic (voice) and facial expressions. The applications of the resulting systems are manifold and range from gaming to indexing and retrieval, through chat and health care. No study has, to the best of our knowledge, ever reported results comparing the effectiveness of several features for automatic emotion recognition. In this work, we present an extensive study conducted on feature selection for automatic, audio-visual, realtime, and person independent emotion recognition. More than 300,000 different neural networks have been trained in order to compare the performances of 64 features and 11 different sets of features with 450 different analysis settings. Results show that: 1) to build an optimal emotion recognition system, different emotions should be classified via different features and 2) different features, in general, require different processing.