Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Data and Features
Discussion and Conclusion
References
All Topics
Computer Science
Computer Vision

Predicting the Valence of a Scene from Observers’ Eye Movements

Profile image of Seppo LaukkaSeppo Laukka

2015, PLOS ONE

https://doi.org/10.1371/JOURNAL.PONE.0138198
visibility

…

description

19 pages

link

1 file

Abstract

Multimedia analysis benefits from understanding the emotional content of a scene in a variety of tasks such as video genre classification and content-based image retrieval. Recently, there has been an increasing interest in applying human bio-signals, particularly eye movements, to recognize the emotional gist of a scene such as its valence. In order to determine the emotional category of images using eye movements, the existing methods often learn a classifier using several features that are extracted from eye movements. Although it has been shown that eye movement is potentially useful for recognition of scene valence, the contribution of each feature is not well-studied. To address the issue, we study the contribution of features extracted from eye movements in the classification of images into pleasant, neutral, and unpleasant categories. We assess ten features and their fusion. The features are histogram of saccade orientation, histogram of saccade slope, histogram of saccade length, histogram of saccade duration, histogram of saccade velocity, histogram of fixation duration, fixation histogram, top-ten salient coordinates, and saliency map. We utilize machine learning approach to analyze the performance of features by learning a support vector machine and exploiting various feature fusion schemes. The experiments reveal that 'saliency map', 'fixation histogram', 'histogram of fixation duration', and 'histogram of saccade slope' are the most contributing features. The selected features signify the influence of fixation information and angular behavior of eye movements in the recognition of the valence of images.

Related papers

The Effect of Eye Movements in Response to Different Types of Scenes Using a Graph-Based Visual Saliency Algorithm
Maria Wahid

Applied Sciences, 2019

Saliency is the quality of an object that makes it stands out from neighbouring items and grabs viewer attention. Regarding image processing, it refers to the pixel or group of pixels that stand out in an image or a video clip and capture the attention of the viewer. Our eye movements are usually guided by saliency while inspecting a scene. Rapid detection of emotive stimuli an ability possessed by humans. Visual objects in a scene are also emotionally salient. As different images and clips can elicit different emotional responses in a viewer such as happiness or sadness, there is a need to measure these emotions along with visual saliency. This study was conducted to determine whether the existing available visual saliency models can also measure emotional saliency. A classical Graph-Based Visual Saliency (GBVS) model is used in the study. Results show that there is low saliency or salient features in sad movies with at least a significant difference of 0.05 between happy and sad videos as well as a large mean difference of 76.57 and 57.0, hence making these videos less emotionally salient. However, overall visual content does not capture emotional salience. The applied Graph-Based Visual Saliency model notably identified happy emotions but could not analyze sad emotions.

View PDFchevron_right
Predicting Eyes’ Fixations in Movie Videos: Visual Saliency Experiments on a New Eye-Tracking Database
Petros Maragos

Lecture Notes in Computer Science, 2014

In this paper we describe the newly created eye tracking annotated database Eye-Tracking Movie Database ETMD and give some preliminary experimental results on this dataset using our new visual saliency frontend. We have developed a database with eye-tracking human annotation that comprises video clips from Hollywood movies, which are longer in duration than the existing databases' videos and include more complex semantics. Our proposed visual saliency frontend is based on both low-level features, such as intensity, color and spatio-temporal energy, and face detection results and provides a single saliency volume map. The described new eye-tracking database can become useful in many applications while our computational frontend shows to be promising as it gave good results on predicting the eye's fixation according to certain metrics.

View PDFchevron_right
Generating Multimodal Features for Emotion Classification from Eye Movement Signals
XIYA GUAN

Aust. J. Intell. Inf. Process. Syst., 2019

In recent years, multimodal signals such as EEG and eye movements have been widely used and achieved a better performance than single modality in emotion classification. However, inconvenience in EEG signals collecting procedure still remains a key problem in practical applications. In this paper, we propose a novel method for generating multimodal features from eye movement signals. As a result, we could simplify classification procedure and use EEG and eye movement signals in training stage and eye movement signals in test stage. We evaluate the effectiveness of our proposed method on classification of five human emotions, which include disgust, fear, sad, neutral and happy. The experimental results indicate that our proposed method could really generate reliable multimodal representations and has a nearly comparable performance (72.80%), with multimodal models (79.70%), while it also outperforms using single eye movement signals (59.66%) and EEG signals (68.58%).

View PDFchevron_right
Gaze differences in proccesing pictures with emotional content
Marijan Palmovic

The International Affective Picture System (IAPS) consists of emotional pictures divided in three categories ; valence, arousal and dominance. This base has been used for different electrophysiological research of emotional processes with no consistent results. One possible explanation for this may be the difference in perceiving pictures. In order to detect what subjects really look at on the presented pictures, an eye-tracking experiment with 12 emotional pictures divided into three categories by valence was conducted. Areas of interest were defined for each picture in order to capture generalizations about how the participants direct their gaze. While perceiving an image, main elements which can define our fixation and exploration on a presented picture are luminosity, color, edges and contrast. In this research black and white pictures were compared with the same pictures in color. Results are interpreted in terms of fixations, saccades and defined areas of interest.

View PDFchevron_right
eSEE-d: Emotional State Estimation Based on Eye-Tracking Dataset
Dimitris Manousos

Brain Sciences

Affective state estimation is a research field that has gained increased attention from the research community in the last decade. Two of the main catalysts for this are the advancement in the data analysis using artificial intelligence and the availability of high-quality video. Unfortunately, benchmarks and public datasets are limited, thus making the development of new methodologies and the implementation of comparative studies essential. The current work presents the eSEE-d database, which is a resource to be used for emotional State Estimation based on Eye-tracking data. Eye movements of 48 participants were recorded as they watched 10 emotion-evoking videos, each of them followed by a neutral video. Participants rated four emotions (tenderness, anger, disgust, sadness) on a scale from 0 to 10, which was later translated in terms of emotional arousal and valence levels. Furthermore, each participant filled three self-assessment questionnaires. An extensive analysis of the parti...

View PDFchevron_right
Automated localization of affective objects and actions in images via caption text-cum-eye gaze analysis
Ramanathan Subramanian

Proceedings of the seventeen ACM international conference on Multimedia - MM '09, 2009

We propose a novel framework to localize and label affective objects and actions in images through a combination of text, visual and gaze-based analysis. Human gaze provides useful cues to infer locations and interactions of affective objects. While concepts (labels) associated with an image can be determined from its caption, we demonstrate localization of these concepts upon learning from a statistical affect model for world concepts. The affect model is derived from non-invasively acquired fixation patterns on labeled images, and guides localization of affective objects (faces, reptiles) and actions (look, read ) from fixations in unlabeled images. Experimental results obtained on a database of 500 images confirm the effectiveness and promise of the proposed approach.

View PDFchevron_right
The influence of emotional picture thematic content on exploratory eye movements
Marcella Caputi

researchgate.net

In picture viewing, emotional vs. neutral stimuli could play a different role in eye movement parameters and in the spatial progression of the scanpath. The aim of this paper is to investigate exploratory behaviour of normal subjects during the vision of emotional vs. non-emotional stimuli, by considering to what extent the thematic content (animate vs. inanimate) is likely to influence the observer's eye movements. Sixty-five subjects' eye movement patterns were measured while looking to emotional (pleasant and unpleasant) and neutral pictures depicting animate or inanimate contents. Results showed that the number of fixations and the gaze duration were greater for emotional pictures than for neutral ones, and animate pictures were fixated longer than inanimate ones. Both emotional and animate pictures may affect eye movements and constitute privileged stimuli of adaptive behavioural tendencies.

View PDFchevron_right
Improving Visual Saliency Computing With Emotion Intensity
Ian Burnett

IEEE transactions on neural networks and learning systems, 2016

Saliency maps that integrate individual feature maps into a global measure of visual attention are widely used to estimate human gaze density. Most of the existing methods consider low-level visual features and locations of objects, and/or emphasize the spatial position with center prior. Recent psychology research suggests that emotions strongly influence human visual attention. In this paper, we explore the influence of emotional content on visual attention. On top of the traditional bottom-up saliency map generation, our saliency map is generated in cooperation with three emotion factors, i.e., general emotional content, facial expression intensity, and emotional object locations. Experiments, carried out on National University of Singapore Eye Fixation (a public eye tracking data set), demonstrate that incorporating emotion does improve the quality of visual saliency maps computed by bottom-up approaches for the gaze density estimation. Our method increases about 0.1 on an avera...

View PDFchevron_right
Exploring Eye Activity as an Indication of Emotional States Using an Eye-Tracking Sensor
Roland Goecke

Studies in Computational Intelligence, 2014

The automatic detection of human emotional states has been of great interest lately for its applications not only in the Human-Computer Interaction field, but also for its applications in psychological studies. Using an emotion elicitation paradigm, we investigate whether eye activity holds discriminative power for detecting affective states. Our emotion elicitation paradigm includes induced emotions by watching emotional movie clips and spontaneous emotions elicited by interviewing participants about emotional events in their life. To reduce gender variability, the selected participants were 60 female native Arabic speakers (30 young adults, and 30 mature adults). In general, the automatic classification results using eye activity were reasonable, giving 66% correct recognition rate on average. Statistical measures show statistically significant differences in eye activity patterns between positive and negative emotions. We conclude that eye activity, including eye movement, pupil dilation and pupil invisibility could be used as a complementary cues for the automatic recognition of human emotional states.

View PDFchevron_right
Recognizing Spontaneous Emotion From The Eye Region Under Different Head Poses
Neethu Narayanan (RCSS)

2018

Emotion recognition is a significant component for efficient person to computer interaction. It plays an acute role in communication, permitting people to express oneself beyond the verbal field. Evolution of emotions from eye expression involves the detection and categorization of various human emotions or state of mind. For instance, in security and surveillance, they can guess the criminal behavior by analyzing the images of their faces from the frames of the video sequence. The main aim of this work is to detect eye emotion of images with various head poses to achieve high accuracy with less complexity. In the existing method, principle component analysis (PCA) feature is extracted and fuzzy clustering technique is used for classification purpose. The accuracy is very low in existing emotion recognition. To overcome this problem, a new emotion recognition technique is introduced in the proposed method. Initially, the viola & jones algorithm is used to detect the face and eye of ...

View PDFchevron_right

References (46)

  1. Yarbus AL. Eye Movements and Vision. Plenum Press; 1967.
  2. Henderson JM, Shinkareva SV, Wang J, Luke SG, Olejarczyk J. Predicting Cognitive State from Eye Movements. PLoS ONE. 2013; 8(5). doi: 10.1371/journal.pone.0064937
  3. Borji A, Itti L. Defending Yarbus: Eye Movements reveal observers' task. Journal of Vision. 2014; 14(5).
  4. Bulling A, Ward JA, Gellersen H, Troster G. Eye Movement Analysis for Activity Recognition Using Electrooculography. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2011; 33(4). doi: 10.1109/TPAMI.2010.86 PMID: 20421675
  5. Subramanian R, Yanulevskaya V, Sebe N. Can computers learn from humans to see better?: inferring scene semantics from viewers' eye movements. In: ACM MM; 2011. p. 33-42.
  6. Isola P, Xiao J, Torralba A, Oliva A. What makes an image memorable? In: CVPR; 2011. p. 145-152.
  7. Mancas M, Meur OL. Memorability of Natural Scenes: The role of Attention. In: ICIP; 2013. p. 196-200.
  8. Le Meur O, Baccino T, Roumy A. Prediction of the Inter-observer Visual Congruency (IOVC) and Appli- cation to Image Ranking. In: ACM MM; 2011. p. 373-382.
  9. Nummenmaa L, Hyönä J, Calvo MG. Eye movement assessment of selective attentional capture by emotional pictures. Emotion. 2006; 6(2). doi: 10.1037/1528-3542.6.2.257 PMID: 16768558
  10. Humphrey K, Underwood G, Lambert T. Salience of the lambs: A test of the saliency map hypothesis with pictures of emotive objects. Journal of Vision. 2012; 12(1). doi: 10.1167/12.1.22 PMID: 22279240
  11. Niu Y, Todd RM, Kyan M, Anderson AK. Visual and emotional salience influence eye movements. ACM Transactions on Applied Perception. 2012; 9(3). doi: 10.1145/2325722.2325726
  12. R Tavakoli H, Yanulevskayay V, Rahtu E, Heikkila J, Sebe N. Emotional Valence Recognition, Analysis of Salience and Eye Movements. In: ICPR; 2014. p. 4666-4671.
  13. Lang PJ, Bradley MM, Cuthbert BN.International affective picture system (IAPS): Affective ratings of pictures and instruction manual. University of Florida, Gainesville, FL; 2008. A-8. Available from: http:// csea.phhp.ufl.edu/media.html.
  14. Ramanathan S, Katti H, Sebe N, Kankanhalli M, Chua TS. An eye fixation database for saliency detec- tion in images. In: ECCV. vol. 6314 of LNCS; 2010. p. 30-43. Available from: http://mmas.comp.nus. edu.sg/NUSEF.html.
  15. Borji A, Tavakoli HR, Sihite DN, Itti L. Analysis of Scores, Datasets, and Models in Visual Saliency Pre- diction. In: ICCV; 2013. p. 921-928.
  16. Kootstra G, Nederveen A, de Boer B. Paying attention to symmetry. In: BMVC; 2008. p. 1115-1125.
  17. Bradley MM, Codispoti M, Sabatinelli D, Lang P. Emotion and motivation II: sex differences in picture processing. Emotion. 2001; 1(3).
  18. Lithari C, Frantzidis CA, Papadelis C, Vivas A, Klados MA, Kourtidou-Papadeli C, et al. Are Females More Responsive to Emotional Stimuli? A Neurophysiological Study Across Arousal and Valence Dimensions. Brain Topography. 2010; 23(1). doi: 10.1007/s10548-009-0130-5 PMID: 20043199
  19. Ma KT, Sim T, Kankanhalli M. VIP: A Unifying Framework for Computational Eye-Gaze Research. In: Salah A, Hung H, Aran O, Gunes H, editors. Human Behavior Understanding. vol. 8212 of LNCS. Springer International Publishing; 2013. p. 209-222.
  20. Wadlinger HA, Isaacowitz DM. Positive mood broadens visual attention to positive stimuli. Motivation and Emotion. 2006; 30. doi: 10.1007/s11031-006-9021-1 PMID: 20431711
  21. van Steenbergen H, Band GPH, Hommel B. Threat But Not Arousal Narrows Attention: Evidence from Pupil Dilation and Saccade Control. Frontiers in Psychology. 2011; 2. doi: 10.3389/fpsyg.2011.00281 PMID: 22059081
  22. Just MA, Carpenter PA. A theory of reading: From eye fixations to comprehension. Psychological Review. 1980; 87. doi: 10.1037/0033-295X.87.4.329 PMID: 7413885
  23. Vitu F, McConkie GW, Kerr P, O'Regan JK. Fixation location effects on fixation durations during read- ing: an inverted optimal viewing position effect. Vision Research. 41;2001. doi: 10.1016/S0042-6989 (01)00166-3 PMID: 11718792
  24. Tichon JG, Mavin T, Wallis G, Visser TAW, Riek S. Using Pupillometry and Electromyography to Track Positive and Negative Affect During Flight Simulation. Aviation Psychology and Applied Human Fac- tors. 2014; 4(1). doi: 10.1027/2192-0923/a000052
  25. Simola J, Fevre KL, Torniainen J, Baccino T. Affective processing in natural scene viewing: Valence and arousal interactions in eye-fixation-related potentials. NeuroImage. 2015; 106(0). PMID: 25463473
  26. Susskind JM, Lee DH, Cusi A, Feiman R, Grabski W, Anderson AK. Expressing fear enhances sensory acquisition. Nature Neuroscience. 2008; 11(7). doi: 10.1038/nn.2138 PMID: 18552843
  27. Chen NM, Clarke PF, Watson TL, MacLeod C, Guastella AJ. Biased Saccadic Responses to Emotional Stimuli in Anxiety: An Antisaccade Study. Plos ONE. 2014; 9(2). doi: 10.1371/journal.pone.0086474
  28. Armstrong T, Olatunji BO. What they see is what you get: Eye tracking of attention in the anxiety disor- ders; 2009. Available from: http://www.apa.org/science/about/psa/2009/03/science-briefs.aspx.
  29. Mikels J, Fredrickson B, Larkin G, Lindberg C, Maglio S, Reuter-Lorenz P. Emotional category data on images from the International Affective Picture System. Behavior Research Methods. 2005; 37(4). doi: 10.3758/BF03192732 PMID: 16629294
  30. Irwin DE. Fixation Location and Fixation Duration as Indices of Cognitive Processing. In: Ferreira F, Henderson JM, editors. The interface of language, vision, and action: Eye movements and the visual world. Psychology Press; 2004. p. 105-133.
  31. Kanan C, Ray N, Bseiso DNF, Hsiao JH, W CG. Predicting an observer's task using multi-fixation pat- tern analysis. In: ACM Symposium on Eye Tracking Research and Applications; 2014. p. 287-290.
  32. Greene MR, Liu T, Wolfe JM. Reconsidering Yarbus: A failure to predict observers' task from eye move- ment patterns. Vision Research. 2012; 62(0). doi: 10.1016/j.visres.2012.03.019 PMID: 22487718
  33. Judd T, Ehinger K, Durand F, Torralba A. Learning to Predict Where Humans Look. In: ICCV; 2009. p. 2106-2113.
  34. Itti L, Koch C, Niebur E. A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence. 1998; 20(11). doi: 10.1109/34.730558
  35. Parkhurst D, Law K, Niebur E. Modeling the role of salience in the allocation of overt visual attention. Vision Research. 2002; 42(1). doi: 10.1016/S0042-6989(01)00250-4 PMID: 11804636
  36. Chen YW, Lin CJ. Combining SVMs with Various Feature Selection Strategies. In: Guyon I, Nikravesh M, Gunn S, Zadeh L, editors. Feature Extraction. vol. 207 of Studies in Fuzziness and Soft Computing. Springer Berlin Heidelberg; 2006. p. 315-324.
  37. Marcano-Cedeno A, Quintanilla-dominguez J, Cortina-Januchs MG, Andina D. Feature selection using Sequential Forward Selection and classification applying Artificial Metaplasticity Neural Network. In: IECON; 2010. p. 2845-2850.
  38. Wahab MNA, Nefti-Meziani S, Atyabi A. A Comprehensive Review of Swarm Optimization Algorithms. PLoS ONE. 2015; 10(5). doi: 10.1371/journal.pone.0122827
  39. Wang L. Support Vector Machines: Theory and Applications. Springer-Verlag Berlin Heidelberg; 2005.
  40. Sanei S, Chambers JA. EEG Signal Processing. John Wiley & Sons, Ltd.; 2007.
  41. Olson DL, Delen D. Advanced Data Mining Techniques. Springer; 2002.
  42. Anderson T. Classification by multivariate analysis. Psychometrika. 1951; 16. doi: 10.1007/ BF02313425
  43. Shahrokh Esfahani M, Dougherty E. Effect of separate sampling on classification accuracy. Bioinfor- matics. 2014; 30(2). doi: 10.1093/bioinformatics/btt662 PMID: 24257187
  44. Powers DMW. Evaluation: From Precision, Recall and F-Measure to ROC., Informedness, Markedness & Correlation. Journal of Machine Learning Technologies. 2011; 2(1).
  45. Mogg K, Millar N, Bradley B. Biases in eye movements to threatening facial expressions in generalized anxiety disorder and depressive disorder. Journal of Abnormal Psychology. 2000; 109(4). doi: 10.1037/ 0021-843X.109.4.695 PMID: 11195993
  46. Goh J, Tan J, Park D. Culture Modulates Eye-Movements to Visual Novelty. PLoS ONE. 2009; 4(12). doi: 10.1371/journal.pone.0008238

Related papers

RESEARCH ARTICLE Predicting the Valence of a Scene from Observers ’ Eye Movements
Samia Nefti

2016

Multimedia analysis benefits from understanding the emotional content of a scene in a vari-ety of tasks such as video genre classification and content-based image retrieval. Recently, there has been an increasing interest in applying human bio-signals, particularly eye move-ments, to recognize the emotional gist of a scene such as its valence. In order to determine the emotional category of images using eye movements, the existing methods often learn a classifier using several features that are extracted from eye movements. Although it has been shown that eye movement is potentially useful for recognition of scene valence, the contribution of each feature is not well-studied. To address the issue, we study the contribu-tion of features extracted from eye movements in the classification of images into pleasant, neutral, and unpleasant categories. We assess ten features and their fusion. The features are histogram of saccade orientation, histogram of saccade slope, histogram of saccad...

View PDFchevron_right
A machine learning approach to predict emotional arousal and valence from gaze extracted features
Dimitris Manousos

2021 IEEE 21st International Conference on Bioinformatics and Bioengineering (BIBE), 2021

In the last years, many studies have been investigating emotional arousal and valence. Most of them have focused on the use of physiological signals such as EEG or EMG, cardiovascular measures or skin conductance. However, eye related features have proven to be very helpful and easy to use metrics, especially pupil size and blink activity. The aim of this study is to predict emotional arousal and valence levels which are induced during emotionally charged situations from eye related features. For this reason, we performed an experimental study where the participants watched emotion-eliciting videos and self-assessed their emotions, while their eye movements were being recorded. In this work, several classifiers such as KNN, SVM, Naive Bayes, Trees and Ensemble methods were trained and tested. Finally, emotional arousal and valence levels were predicted with 85 and 91% efficiency, respectively.

View PDFchevron_right
Eye-Tracking Analysis for Emotion Recognition
Marcin Kołodziej

is article reports the results of the study related to emotion recognition by using eye-tracking. Emotions were evoked by presenting a dynamic movie material in the form of 21 video fragments. Eye-tracking signals recorded from 30 participants were used to calculate 18 features associated with eye movements (fixations and saccades) and pupil diameter. To ensure that the features were related to emotions, we investigated the influence of luminance and the dynamics of the presented movies. ree classes of emotions were considered: high arousal and low valence, low arousal and moderate valence, and high arousal and high valence. A maximum of 80% classification accuracy was obtained using the support vector machine (SVM) classifier and leave-one-subject-out validation method.

View PDFchevron_right
Eye gaze patterns in emotional pictures
Enzo Scilingo, Antonio Lanatà

Journal of Ambient Intelligence and Humanized Computing, 2013

This paper reports on a preliminary study aiming at investigating the eye gaze pattern and pupil size variation to discriminate emotional states induced by looking at pictures having different arousal content. A wearable and wireless eye gaze tracking system, hereinafter called HATCAM, which was able to robustly detect eye tracking and pupil area was used. A group of ten volunteers was presented with a set of neutral and arousal pictures extracted from the International Affective Picture System according to an ad-hoc experimental protocol. A set of features was extracted from eye gaze patterns and pupil size variations and used to classify the two classes of pictures. Although preliminary, results are very promising for affective computing applications.

View PDFchevron_right
Evaluation of Human Emotion from Eye Motions
Vidas Raudonis, Gintaras Dervinis, Andrius Vilkauskas

International Journal of Advanced Computer Science and Applications, 2013

The object of this paper is to develop an emotion recognition system that analysis the motion trajectory of the eye and gives the response on appraisal emotion. The emotion recognition solution is based on the data gathering using head mounted eye tracking device. The participants of experimental investigation were provided with a visual stimulus (PowerPoint slides) and the emotional feedback was determined by the combination of eye tracking device and emotion recognition software. The stimulus was divided in four groups by the emotion that should be triggered in the human, i.e., neutral, disgust, exhilaration and excited. Some initial experiments and the data on the recognition accuracy of the emotion from eye motion trajectory are provided along with the description of implemented algorithms.

View PDFchevron_right

Related topics

  • Computer Science
  • Artificial Intelligence
  • Computer Vision
  • Medicine
  • Computer User Interface Design
  • Observer Variation
  • Histogram matching
  • photic stimulation
  • Histogram of oriented gradients

    • Cited by

    Neural Correlates of Preference: A Transmodal Validation Study
    Alvaro M Dias

    Frontiers in Human Neuroscience

    Liking is one of the most important psychological processes associated with the reward system, being involved in affective processing and pleasure/displeasure encoding. Currently, there is no consensus regarding the combination of physiological indicators which best predict liking, especially when applied to dynamic stimuli such as videos. There is a lack of a standard methodology to assess likeability over time and therefore in assessing narrative and semantic aspects of the stimulus. We developed a time-dependent method to evaluate the physiological correlates of likeability for three different thematic categories, namely: adventure (AV), comedy (CM), and nature landscape (LS). Twenty-eight healthy adults with ages ranging from 18 to 35 years (average: 23.85 years) were enrolled in the study. The participants were asked to provide likeability ratings for videos as they watched them, using a response box. Three 60-s videos were presented, one for each category, in randomized order while the participant's physiological data [electroencephalogram (EEG), electrocardiogram (ECG) and eye tracking (ET)] was recorded. The comedy video (CM) presented the smallest minimum accumulated normalized rating (ANR; p = 0.013) and the LS video presented the highest maximum ANR (p = 0.039). The LS video presented the longest time for first response (p < 0.001) and the AV video presented the shortest time for maximum response (p = 0.016). The LS video had the highest mean likeability rating with 1.43 ± 2.31 points; and the CM video had the lowest with 0.57 ± 1.77. Multiple linear regression models were created to predict the likeability of each video using the following physiological indicators; AV: power in beta band at C4 and P4 (p = 0.004, adj. R 2 = 0.301); CM: alpha power in Fp2 (p = 0.001, adj. R 2 = 0.326) and LS: alpha power in P4, F8, and Fp2; beta power in C4 and P4 and pupil size, (p = 0.002, adj. R 2 = 0.489). Despite its limitations (e.g., using one 1-min video per category) our findings suggest that there is a considerable difference in the psychophysiological correlates of stimuli with different contextual properties and that the use of time-dependent methods to assess videos should be considered as best practices.

    View PDFchevron_right
    Gender and emotion recognition with implicit user signals
    Seyed Mostafa Kia

    2017

    We examine the utility of implicit user behavioral signals captured using low-cost, off-the-shelf devices for anonymous gender and emotion recognition. A user study designed to examine male and female sensitivity to facial emotions confirms that females recognize (especially negative) emotions quicker and more accurately than men, mirroring prior findings. Implicit viewer responses in the form of EEG brain signals and eye movements are then examined for existence of (a) emotion and gender-specific patterns from event-related potentials (ERPs) and fixation distributions and (b) emotion and gender discriminability. Experiments reveal that (i) Gender and emotion-specific differences are observable from ERPs, (ii) multiple similarities exist between explicit responses gathered from users and their implicit behavioral signals, and (iii) Significantly above-chance (≈70%) gender recognition is achievable on comparing emotion-specific EEG responses– gender differences are encoded best for a...

    View PDFchevron_right
    Computational modeling of human reasoning processes for interpretable visual knowledge: a case study with radiographers
    Sanda Erdelez

    Scientific Reports, 2020

    Visual reasoning is critical in many complex visual tasks in medicine such as radiology or pathology. It is challenging to explicitly explain reasoning processes due to the dynamic nature of real-time human cognition. A deeper understanding of such reasoning processes is necessary for improving diagnostic accuracy and computational tools. Most computational analysis methods for visual attention utilize black-box algorithms which lack explainability and are therefore limited in understanding the visual reasoning processes. In this paper, we propose a computational method to quantify and dissect visual reasoning. The method characterizes spatial and temporal features and identifies common and contrast visual reasoning patterns to extract significant gaze activities. The visual reasoning patterns are explainable and can be compared among different groups to discover strategy differences. Experiments with radiographers of varied levels of expertise on 10 levels of visual tasks were cond...

    View PDFchevron_right
    Exploring the Usage of Commercial Bio-Sensors for Multitasking Detection
    Yasmeen Abdrabou

    Proceedings of the 17th International Conference on Mobile and Ubiquitous Multimedia, 2018

    Most of the current adaptive systems support single task activities. The rise in the number of daily interactive devices and sources of information made multitasking an integral activity in our daily life. Affect-aware systems show exciting potential to support the user, however, they focus on the induced effect of an additional task in terms of cognitive load and stress, rather than the influence of the number of tasks i.e. multitasking. This paper presents indicators of the number of tasks being performed by the user using a set of bio-sensors. A preliminary user study was conducted with two follow-up explorations. Our findings imply that we can distinguish between the number of tasks performed based on high-end as well as cheap Heart Rate sensors. Additionally, tasks number correlates with other signals, namely wrist and forehead temperature. We provide empirical evidence showing how to differentiate between single-and dual-tasking activities.

    View PDFchevron_right
    580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved