Detecting head movements in video-recorded dyadic conversations

2020

This paper presents an approach to automatic head movement detection and classification in data from a corpus of video-recorded face-to-face conversations in Danish involving 12 different speakers. A number of classifiers were trained with different combinations of visual, acoustic and word features and tested in a leave-one-out cross validation scenario. The visual movement features were extracted from the raw video data using OpenPose, and the acoustic ones using Praat. The best results were obtained by a Multilayer Perceptron classifier, which reached an average 0.68 F1 score across the 12 speakers for head movement detection, and 0.40 for head movement classification given four different classes. In both cases, the classifier outperformed a simple most frequent class baseline as well as a more advanced baseline only relying on velocity features.

Proceedings of the Sixth Workshop on Vision and Language, 2017

We motivate and describe a new freely available human-human dialogue data set for interactive learning of visually grounded word meanings through ostensive definition by a tutor to a learner. The data has been collected using a novel, character-by-character variant of the DiET chat tool (Healey et al., 2003; Mills and Healey, submitted) with a novel task, where a Learner needs to learn invented visual attribute words (such as "burchak" for square) from a tutor. As such, the text-based interactions closely resemble face-to-face conversation and thus contain many of the linguistic phenomena encountered in natural, spontaneous dialogue. These include self-and other-correction, mid-sentence continuations, interruptions, overlaps, fillers, and hedges. We also present a generic n-gram framework for building user (i.e. tutor) simulations from this type of incremental data, which is freely available to researchers. We show that the simulations produce outputs that are similar to the original data (e.g. 78% turn match similarity). Finally, we train and evaluate a Reinforcement Learning dialogue control agent for learning visually grounded word meanings, trained from the BURCHAK corpus. The learned policy shows comparable performance to a rulebased system built previously.

— Human head gestures convey a rich message, containing information deliver for peoples as a communication tool. Nodding, shacking are commonly used gestures as non-verbal signals to communicate their intent and emotions. However, the majority of head gestures classification systems focused on head nodding and shaking detection. while they ignored other head gestures which have more expressive emotional signals like rest(up and down), turn, tilt, and tilting. In this paper we developed a new model to classify all head gestures (rest, turn, tilt, node, shake, and tilting) from complex head motions. The model methodology based on distinguishing basic head movements (rest, turn, and tilt) and combined movements (nodding, shaking, and tilting). The purpose of this system is to detect and label combined and basic head movements in dynamic video. In addition, this phase of this study looking at developing an affective machine uses head movements to extract complex affective states (this work is underway). The system used 3D head rotation angles to classify relevant head gestures in-plan and out-plan of view during user interaction with computer. This system used an open source tracker to detect and track head movements. The Three angels that obtained from the tracker (pitch, yaw, and roll), were analyzed and packed into sequences of observation symbols or cues. Observations formed inputs to an all-vs-all discrete Hidden Markov Model (HMM) classifier. Three classifiers were used for each angle. The classifiers are trained on Boston University dataset, and tested on available mind reading data. The system evaluate on video streams in real time by webcam. The system is fully automatic without incurring any cost of technical methods and doesn't require any sensitive tools. I. INTRODUCTION Human head gestures usually attempt to convey a message which carries out different meanings. The can be considered as a communication tool, particularly in conversations. For instance, head nodding gesture during a conversation is a sign of agreement or acceptance, that is to say, a pointer to listener attention, and understanding. Also, head shaking gesture refers to conceptual disagreement [1] [2]. The ability to detect head motion by people is an effortless task [2]. However, it deemed difficult and challenging task in both computer vision context and human-computer interaction. Computational head poses estimation requires extensive processing to infer head rotation and translation. These transformations can be either taken with respect to a camera or world coordinates [3]. The existing literature on head gestures as a tool is extensive and focuses particularly on detecting head nod and shake gestures. Therefore, the majority of works investigated in this paper focused on the aforementioned points. While other head gestures get less attention, although it can observe specific emotions, for example head up movement related with pride, boredom and contempt, head down linked with greeting, guilt, embarrassment, and shame [4]. The overall aim of this paper is to develop a solid and robust system based on 3DOF, by using All-vs-All HMM classification to categorize combined head gestures (i.e., nod, shake, and tilting) and basic gestures (rest, turn, and tilt) in real time. This approach uses an open source head tracker and a webcam. The head tracking model is based on 3D head model aligned with 68 detected face feature points to detect head orientation (pitch, yaw, and roll). Other researchers used methods based on the coordinates of face feature points like (eye corner or nose tip) and use geometric calculations to estimate head orientation. The limitations of using features points are in incidents of occlusion due to changes in illumination. This system performs in an uncontrolled environment with variable illumination. Also, it does not assume that the head should start with neutral pose against other approaches. The rest paper structure as follow; in section I we introduce a brief review of related works and important methods which used in head tracking and detection. Section III describes the methodology used for head tracking and extracting head actions/cues, which then used as observations in HMM. Section IV presents general review of HMM model and its elements with its problems. Model training methods can be found in section V and Classification methods in section VI. Results and system evaluation can be found in section VII and section VIII is for conclusions.

2014

This paper introduces a multimodal discussion corpus for the study into head movement and turn-taking patterns in debates. Given that participants either acted alone or in a pair, cooperation and competition and their nonverbal correlates can be analyzed. In addition to the video and audio of the recordings, the corpus contains automatically estimated head movements, and manual annotations of who is speaking and who is looking where. The corpus consists of over 2 hours of debates, in 6 groups with 18 participants in total. We describe the recording setup and present initial analyses of the recorded data. We found that the person who acted as single debater speaks more and also receives more attention compared to the other debaters, also when corrected for the time speaking. We also found that a single debater was more likely to speak after a team debater. Future work will be aimed at further analysis of the relation between speaking and looking patterns, the outcome of the debate an...

IEEE Access, 2020

A functional head-movement corpus and convolutional neural networks (CNNs) for detecting head-movement functions are presented for analyzing the multiple communicative functions of head movements in multiparty face-to-face conversations. First, focusing on the multifunctionality of head movements, i.e., that a single head movement can simultaneously perform multiple functions, this paper defines 32 nonmutually-exclusive function categories, whose genres are speech production, eliciting and giving feedback, turn management, and cognitive and affect display. To represent and capture arbitrary multifunctional structures, our corpus employs multiple binary codes and logical-sum-based aggregations of multiple coders' judgments. A corpus analysis targeting four-party Japanese conversations revealed multifunctional patterns in which the speaker modulates multiple functions, such as emphasis and eliciting listeners' responses, through rhythmic head movements, and listeners express various attitudes and responses through continuous back-channel head movements. This paper proposes CNN-based binary classifiers for detecting each of the functions from the angular velocity of the head pose and the presence or absence of utterances. The experimental results showed that the recognition performance varies greatly, from approximately 30% to 90% in terms of the F-score, depending on the function category, and the performance was positively correlated with the amount of data and inter-coder agreement. In addition, we noted a tendency toward overdetection that added more functions to those originally in the corpus. The analyses and experiments confirm that our approach is promising for studying the multifunctionality of head movements. INDEX TERMS Deep neural networks, gesture recognition, meeting analysis, multimodal sensor, nonverbal behaviors, social signal processing.