Smart Doll: Emotion Recognition Using Embedded Deep Learning

International Journal of Advanced Computer Science and Applications, 2021

The need for an efficient intelligent system to detect human emotions is imperative. In this study, we proposed an automated convolutional neural network-based approach to recognize the human mental state from eyes and their surrounding features. We have applied deep convolutional neural network based Keras applications with the help of transfer learning and fine-tuning. We have worked with six universal emotions (i.e., happiness, disgust, sadness, fear, anger, and surprise) with a dataset containing 588 unique double eye images. In this study, we considered the eyes and their surrounding areas (Upper and lower eyelid, glabella, and brow) to detect the emotional state. The state and movement of the iris and pupil can vary with the various mental states. The common features found within the entire eyes during different mental states can help to capture human expression. The dataset was trained with pretrained weights and used a confusion matrix to analyze the prediction to achieve better accuracy. The highest accuracy was achieved by DenseNet-201 is 91.78%, whereas VGG-16 and Inception-ResNet-v2 show 90.43% and 89.67%, respectively. This study will provide an insight into the current state of research to obtain better facial recognition.

IJEER , 2022

This paper discusses a deep learning-based image processing method to recognize human emotion from their facial expression with special concentration on infant’s face between one to five years of age. The work has importance because most of the time it becomes necessary to understand need of a child from their facial expression and behavior. This work is still a challenge in the field of Human Facial Emotion Recognition due to confusing facial expression that sometimes found in some of the samples. We have tried to recognize any facial expression into one of the mostly understood human mood namely Angry, Disgust, Fear, Happy, Sad, Surprise and Neutral. For this purpose, we have trained an image classifier with Convolutional Neural Network with Kaggle's Fer2013 Dataset. After the completion of the project, we achieved good accuracy in most of the prominent emotions by testing with 20 random images for each emotion.

International Journal of Scientific Research in Computer Science, Engineering and Information Technology, 2023

Emotions are mental states that accompany physiological changes in the face, resulting in facial expressions. Sympathy, anger, worry, joy, fright, and other significant emotions are a few examples. Facial expressions play a significant role in non-verbal communication because they encapsulate a person's emotions. There has been a great deal of research done on computer modelling of human emotions. Computer modelling of human emotions has been made possible by computer technology. However, it is still in its infancy. The authors attempted to overcome limitations and create new opportunities as well as gain a better understanding and implement this simple form of human interaction in proposed computer-aided world. It has been made possible to evaluate and interpret genuine facial expressions in real time thanks to new techniques for collecting facial expressions and quick, highresolution pictures. The FER (Facial Expression Recognition) method currently relies on motionless frames, which makes it very hard to recognize foreground from background in the absence of motion information. This study describes a real-time facial expression identification system that detects faces using HAAR cascading classification and classifies facial expressions using convolutional neural networks. The system utilizes a webcam to dynamically display emotion text and accurately categorizes seven major emotions, including anger, disgust, fear, happiness, sadness, surprise, and neutrality. Real-time facial expression recognition may be utilised in a number of real-world applications, including as airport security, trade, and medical monitoring.

Facial emotion recognition (FER) forms part of affective computing, where computers are trained to recognize human emotion from human expressions. Facial Emotion Recognition is very necessary for bridging the communication gap between humans and computers because facial expressions are a form of communication that transmits 55% of a person's emotional and mental state in a total face-to-face communication spectrum. Breakthroughs in this field also make computer systems (robotic systems) better serve or interact with humans. Research has far advanced for this cause, and Deep learning is at its heart. This paper systematically discusses state-of-the-art deep learning architectures and algorithms for facial emotion detection and recognition. The paper also reveals the dominance of CNN architectures over other known architectures like RNNs and SVMs, highlighting the contributions, model performance, and limitations of the reviewed state-ofthe-art. It further identifies available opportunities and open issues worth considering by various FER research in the future. This paper will also discover how computation power and availability of large facial emotion datasets have also limited the pace of progress.

2017 IEEE International Conference on Consumer Electronics (ICCE), 2017

By growing the capacity and processing power of the handheld devices nowadays, a wide range of capabilities can be implemented in these devices to make them more intelligent and user friendly. Determining the mood of the user can be used in order to provide suitable reactions from the device in different conditions. One of the most studied ways of mood detection is by using facial expressions, which is still one of the challenging fields in pattern recognition and machine learning science. Deep Neural Networks (DNN) have been widely used in order to overcome the difficulties in facial expression classification. In this paper it is shown that the classification accuracy is significantly lower when the network is trained with one database and tested with a different database. A solution for obtaining a general and robust network is given as well.

ArXiv, 2018

While several approaches to face emotion recognition task are proposed in literature, none of them reports on power consumption nor inference time required to run the system in an embedded environment. Without adequate knowledge about these factors it is not clear whether we are actually able to provide accurate face emotion recognition in the embedded environment or not, and if not, how far we are from making it feasible and what are the biggest bottlenecks we face. The main goal of this paper is to answer these questions and to convey the message that instead of reporting only detection accuracy also power consumption and inference time should be reported as real usability of the proposed systems and their adoption in human computer interaction strongly depends on it. In this paper, we identify the state-of-the art face emotion recognition methods that are potentially suitable for embedded environment and the most frequently used datasets for this task. Our study shows that most o...

Research Square (Research Square), 2023

The study of facial emotion recognition (FER) holds signi cant importance within the realm of academic research, since it has wide-ranging rami cations across multiple areas such as mental health assessment and human-computer interaction. This work introduces a novel methodology for FER that integrates Gabor lter-based feature extraction with a deep Convolutional Neural Network (CNN). The utilisation of Gabor lters enables extraction of prominent textural characteristics from facial images, whilst CNNs acquire informative representations to achieve precise emotion classi cation. The proposed methodology is assessed using the FER2013 dataset and compared with pre-existing methodologies. The ndings illustrate the e cacy of our methodology in accurately identifying facial expressions of emotions, emphasising its potential for practical implementation in the elds of mental health research and emotion-sensitive systems. The method demonstrates improved accuracy and resilience by combining Gabor lters and CNNs, showing potential for enhancing mental health evaluation and enabling adaptive human-computer interaction. This study makes a valuable contribution to the elds of health, mental health, and adaptation by advancing the creation of emotion-aware technologies that are designed to address the unique emotional requirements of individuals.

2017 Seventh International Conference on Innovative Computing Technology (INTECH), 2017

From the last decade, researches on human facial emotion recognition disclosed that computing models built on regression modelling can produce applicable performance. However, many systems need extensive computing power to be run that prevents its wide applications such as robots and smart devices. In this proposed system, a real-time automatic facial expression system was designed, implemented and tested on an embedded device such as FPGA that can be a first step for a specific facial expression recognition chip for a social robot. The system was built and simulated in MATLAB and then was built on FPGA and it can carry out real time continuously emotional state recognition at 30 fps with 47.44% accuracy. The proposed graphic user interface is able to display the participant video and two dimensional predict labels of the emotion in real time together.

Advances in Human-Computer Interaction

Nowadays, much research attention is focused on human–computer interaction (HCI), specifically in terms of biosignal, which has been recently used for the remote controlling to offer benefits especially for disabled people or protecting against contagions, such as coronavirus. In this paper, a biosignal type, namely, facial emotional signal, is proposed to control electronic devices remotely via emotional vision recognition. The objective is converting only two facial emotions: a smiling or nonsmiling vision signal captured by the camera into a remote control signal. The methodology is achieved by combining machine learning (for smiling recognition) and embedded systems (for remote control IoT) fields. In terms of the smiling recognition, GENKl-4K database is exploited to train a model, which is built in the following sequenced steps: real-time video, snapshot image, preprocessing, face detection, feature extraction using HOG, and then finally SVM for the classification. The achieve...