Expert system for automatic analysis of facial expressions

Lecture Notes in Computer Science, 2012

In the present work a cross-modal evaluation of the visual and auditory channels in conveying emotional information is conducted through perceptual experiments aimed at investigating whether some of the basic emotions are perceptually privileged and whether the perceptual mode, the cultural environment and the language play a role in this preference. To this aim, Hungarian subjects were requested to assess emotional stimuli extracted from Italian and American movies in the single (either mute video or audio alone) and combined audio-video mode. Results showed that among the proposed emotions, anger plays a special role and fear, happiness and sadness are better perceived than surprise and irony in both the cultural environments. The perception of emotions is affected by the communication mode and the language influences the perceptual assessment of emotional information.

International Journal of Pattern Recognition and Artificial Intelligence, 2009

Human face analysis has attracted a large number of researchers from various fields, such as computer vision, image processing, neurophysiology or psychology. One of the particular aspects of human face analysis is encompassed by facial expression recognition task. A novel method based on phase congruency for extracting the facial features used in the facial expression classification procedure is developed. Considering a set of image samples comprising humans expressing various expressions, this new approach computes the phase congruency map between the samples. The analysis is performed in the frequency space where the similarity (or dissimilarity) between sample phases is measured to form discriminant features. The experiments were run using samples from two facial expression databases. To assess the method's performance, the technique is compared to the state-of-the art techniques utilized for classifying facial expressions, such as Principal Component Analysis (PCA), Independent Component Analysis (ICA), Linear Discriminant Analysis (LDA), and Gabor jets. The features extracted by the aforementioned techniques are further classified using two classifiers: a distance-based classifier and a Support Vector Machine -based classifier. Experiments reveal superior facial expression recognition performance for the proposed approach with respect to other techniques.

Lecture Notes in Computer Science, 2007

Two unobtrusive modalities for automatic emotion recognition are discussed: speech and facial expressions. First, an overview is given of emotion recognition studies based on a combination of speech and facial expressions. We will identify difficulties concerning data collection, data fusion, system evaluation and emotion annotation that one is most likely to encounter in emotion recognition research. Further, we identify some of the possible applications for emotion recognition such as health monitoring or e-learning systems. Finally, we will discuss the growing need for developing agreed standards in automatic emotion recognition research.

The Imaging Science Journal, 2015

The authors in this study proposed facial expression recognition system in order to improve the expression recognition performance over the recently proposed systems. Feature sets for all training samples are constructed based on speed up robust feature descriptors. An ensemble of feature sets is then created incrementally. To achieve high diversity of ensemble, the dissimilarities between the training samples for each class are computed. This high diversity led to a high recognition rate. Experimentation on two publicly available datasets is performed. The system achieved 98?6% accuracy on JAFFE dataset and 96?3% accuracy on Multimedia Understanding Group dataset. The results of proposed system are compared with recently proposed work in this area and proved the soundness of the proposed method.

TheScientificWorldJournal, 2014

We report current findings when considering video recordings of facial expressions and body movements to provide affective personalized support in an educational context from an enriched multimodal emotion detection approach. In particular, we describe an annotation methodology to tag facial expression and body movements that conform to changes in the affective states of learners while dealing with cognitive tasks in a learning process. The ultimate goal is to combine these annotations with additional affective information collected during experimental learning sessions from different sources such as qualitative, self-reported, physiological, and behavioral information. These data altogether are to train data mining algorithms that serve to automatically identify changes in the learners' affective states when dealing with cognitive tasks which help to provide emotional personalized support.

Proceedings of the 19th ACM International Conference on Multimodal Interaction, 2017

This paper presents a new approach for detecting pain in sequences of spontaneous facial expressions. The motivation for this work is to accompany mobile-based self-management of chronic pain as a virtual sensor for tracking patients' expressions in real-world settings. Operating under such constraints requires a resource efficient approach for processing non-posed facial expressions from unprocessed temporal data. In this work, the facial action units of pain are modeled as sets of distances among related facial landmarks. Using standardized measurements of pain versus no-pain that are specific to each user, changes in the extracted features in relation to pain are detected. The activated features in each frame are combined using an adapted form of the Prkachin and Solomon Pain Intensity scale (PSPI) to detect the presence of pain per frame. Painful features must be activated in N consequent frames (time window) to indicate the presence of pain in a session. The discussed method was tested on 171 video sessions for 19 subjects from the McMaster painful dataset for spontaneous facial expressions. The results show higher precision than coverage in detecting sequences of pain. Our algorithm achieves 94% precision (F-score=0.82) against human observed labels, 74% precision (F-score=0.62) against automatically generated pain intensities and 100% precision (F-score=0.67) against self-reported pain intensities.

EURASIP Journal on Image and Video Processing, 2007

This paper presents a robust and adaptable facial feature extraction system used for facial expression recognition in humancomputer interaction (HCI) environments. Such environments are usually uncontrolled in terms of lighting and color quality, as well as human expressivity and movement; as a result, using a single feature extraction technique may fail in some parts of a video sequence, while performing well in others. The proposed system is based on a multicue feature extraction and fusion technique, which provides MPEG-4-compatible features assorted with a confidence measure. This confidence measure is used to pinpoint cases where detection of individual features may be wrong and reduce their contribution to the training phase or their importance in deducing the observed facial expression, while the fusion process ensures that the final result regarding the features will be based on the extraction technique that performed better given the particular lighting or color conditions. Real data and results are presented, involving both extreme and intermediate expression/emotional states, obtained within the sensitive artificial listener HCI environment that was generated in the framework of related European projects.

Multimedia Tools and Applications, 2018

Facial Expression Recognition (FER) is inherently data driven. Spontaneous expressions are substantially different from posed expressions. Spontaneous facial expressions are more challenging and more difficult to recognize. Any facial expression can be represented in many different patterns of muscles movements. Moreover, the facial expressions show discrepancies in different cultures and ethnicities. Therefore, a FER system has to learn a huge problem/feature space. A base classifier trained on a sub-region of feature space, cannot perform equally well in other areas of feature space. Therefore, a base classifier should be assigned higher voting weight in the regions near to its training space and a lower voting weight in the regions far from its training space. In order to maintain high accuracy and robustness of a FER system in space and time, a Dynamic Weight Majority Voting (DWMV) mechanism for base classifiers is introduced. An ensemble system for FER is proposed that has the aptitude of incrementally learning and thus, can learn all possible patterns of expressions that may be generated in feature or in various cultures and ethnicities. Speeded-Up Robust Features (SURF) are used to represent the feature space. Since no work in literature is found on Bwhich similarity measure is more appropriate in SURF descriptor domain for facial expression recognition^, therefore, different similarity measures are used and the results are Multimed Tools Appl

PLOS ONE

Background and objective Patients suffering from Parkinson’s disease (PD) present a reduction in facial movements called hypomimia. In this work, we propose to use machine learning facial expression analysis from face images based on action unit domains to improve PD detection. We propose different domain adaptation techniques to exploit the latest advances in automatic face analysis and face action unit detection. Methods Three different approaches are explored to model facial expressions of PD patients: (i) face analysis using single frame images and also using sequences of images, (ii) transfer learning from face analysis to action units recognition, and (iii) triplet-loss functions to improve the automatic classification between patients and healthy subjects. Results Real face images from PD patients show that it is possible to properly model elicited facial expressions using image sequences (neutral, onset-transition, apex, offset-transition, and neutral) with accuracy improvem...

International Journal of Electronics, Communications, and Measurement Engineering, 2019

Facial feature extraction and recognition plays a prominent role in human non-verbal interaction and it is one of the crucial factors among pose, speech, facial expression, behaviour and actions which are used in conveying information about the intentions and emotions of a human being. In this article an extended local binary pattern is used for the feature extraction process and a principal component analysis (PCA) is used for dimensionality reduction. The projections of the sample and model images are calculated and compared by Euclidean distance method. The combination of extended local binary pattern and PCA (ELBP+PCA) improves the accuracy of the recognition rate and also diminishes the evaluation complexity. The evaluation of proposed facial expression recognition approach will focus on the performance of the recognition rate. A series of tests are performed for the validation of algorithms and to compare the accuracy of the methods on the JAFFE, Extended Cohn-Kanade images da...