Vocal Expression and Perception of Emotion

Philips Research Book Series, 2010

Journal on Multimodal User Interfaces, 2013

Recording and annotating a multimodal database of natural expressivity is a task that requires careful planning and implementation, before even starting to apply feature extraction and recognition algorithms. Requirements and characteristics of such databases are inherently different than those of acted behaviour, both in terms of unconstrained expressivity of the human participants, and in terms of the expressed emotions. In this paper, we describe a method to induce, record and annotate natural emotions, which was used to provide multimodal data for dynamic emotion recognition from facial expressions and speech prosody; results from a dynamic recognition algorithm, based on recurrent neural networks, indicate that multimodal processing surpasses both speech and visual analysis by a wide margin. The SAL database was used in the framework of the Humaine Network of Excellence as a common ground for research in everyday, natural emotions.

PLoS ONE, 2014

A total laryngectomy removes the vocal folds which are fundamental in forming voiced sounds that make speech possible. Although implanted prosthetics are commonly used in developed countries, simple handheld vibrating electrolarynxes are still common worldwide. These devices are easy to use but suffer from many drawbacks including dedication of a hand, mechanical sounding voice, and sound leakage. To address some of these drawbacks, we introduce a novel electrolarynx that uses vibro-acoustic interference of dual ultrasonic waves to generate an audible fundamental frequency. A 3D simulation of the principles of the device is presented in this paper.

Trends in hearing

The question of how hearing loss and hearing rehabilitation affect patients' momentary emotional experiences is one that has received little attention but has considerable potential to affect patients' psychosocial function. This article is a product from the Hearing, Emotion, Amplification, Research, and Training workshop, which was convened to develop a consensus document describing research on emotion perception relevant for hearing research. This article outlines conceptual frameworks for the investigation of emotion in hearing research; available subjective, objective, neurophysiologic, and peripheral physiologic data acquisition research methods; the effects of age and hearing loss on emotion perception; potential rehabilitation strategies; priorities for future research; and implications for clinical audiologic rehabilitation. More broadly, this article aims to increase awareness about emotion perception research in audiology and to stimulate additional research on th...

Neural plasticity, 2007

A phylogenetic approach to anxiety is proposed. The different facets of human anxiety and their presence at different levels of the phylum are examined. All organisms, including unicellular such as protozoan, can display a specific reaction to danger. The mechanisms enabling the appraisal of harmful stimuli are fully present in insects. In higher invertebrates, fear is associated with a specific physiological response. In mammals, anxiety is accompanied by specific cognitive responses. The expression of emotions diversifies in higher vertebrates, only primates displaying facial expressions. Finally, autonoetic consciousness, a feature essential for human anxiety, appears only in great apes. This evolutive feature parallels the progress in the complexity of the logistic systems supporting it (e.g., the vegetative and central nervous systems). The ability to assess one's coping potential, the diversification of the anxiety responses, and autonoetic consciousness seem relevant mark...

Scientific Reports

Studies of acoustic communication often focus on the categories and units of vocalizations, but subtle variation also occurs in how these signals are uttered. In human speech, it is not only phonemes and words that carry information but also the timbre, intonation, and stress of how speech sounds are delivered (often referred to as “paralinguistic content”). In non-human animals, variation across utterances of vocal signals also carries behaviorally relevant information across taxa. However, the discriminability of these cues has been rarely tested in a psychophysical paradigm. Here, we focus on acoustic communication in the zebra finch (Taeniopygia guttata), a songbird species in which the male produces a single stereotyped motif repeatedly in song bouts. These motif renditions, like the song repetitions of many birds, sound very similar to the casual human listener. In this study, we show that zebra finches can easily discriminate between the renditions, even at the level of singl...

Journal of Nonverbal Behavior

Past research has largely focused on how emotional expressions provide information about the speaker’s emotional state, but has generally neglected vocal affect’s influence over communication effectiveness. This is surprising given that other nonverbal behaviors often influence communication between individuals. In the present theory paper, we develop a novel perspective called the Contextual Influences of Vocal Affect (CIVA) model to predict and explain the psychological processes by which vocal affect may influence communication through three broad categories of process: emotion origin/construal, changing emotions, and communication source inferences. We describe research that explores potential moderators (e.g., affective/cognitive message types, message intensity), and mechanisms (e.g., emotional assimilation, attributions, surprise) shaping the effects of vocally expressed emotions on communication. We discuss when and why emotions expressed through the voice can influence the ...

Psychological Science, 2014

A central question in the study of human behavior is whether certain emotions, such as anger, fear, and sadness, are recognized in nonverbal cues across cultures. We predicted and found that in a concept-free experimental task, participants from an isolated cultural context (the Himba ethnic group from northwestern Namibia) did not freely label Western vocalizations with expected emotion terms. Responses indicate that Himba participants perceived more basic affective properties of valence (positivity or negativity) and to some extent arousal (high or low activation). In a second, concept-embedded task, we manipulated whether the target and foil on a given trial matched in both valence and arousal, neither valence nor arousal, valence only, or arousal only. Himba participants achieved above-chance accuracy only when foils differed from targets in valence only. Our results indicate that the voice can reliably convey affective meaning across cultures, but that perceptions of emotion fr...

Cochlear Implants International, 2013

Objectives: This study examined the ability of prelingually deaf children with bilateral implants to identify emotion (i.e. happiness or sadness) in speech and music. Methods: Participants in Experiment 1 were 14 prelingually deaf children from 5-7 years of age who had bilateral implants and 18 normally hearing children from 4-6 years of age. They judged whether linguistically neutral utterances produced by a man and woman sounded happy or sad. Participants in Experiment 2 were 14 bilateral implant users from 4-6 years of age and the same normally hearing children as in Experiment 1. They judged whether synthesized piano excerpts sounded happy or sad. Results: Child implant users' accuracy of identifying happiness and sadness in speech was well above chance levels but significantly below the accuracy achieved by children with normal hearing. Similarly, their accuracy of identifying happiness and sadness in music was well above chance levels but significantly below that of children with normal hearing, who performed at ceiling. For the 12 implant users who participated in both experiments, performance on the speech task correlated significantly with performance on the music task and implant experience was correlated with performance on both tasks. Discussion: Child implant users' accurate identification of emotion in speech exceeded performance in previous studies, which may be attributable to fewer response alternatives and the use of child-directed speech. Moreover, child implant users' successful identification of emotion in music indicates that the relevant cues are accessible at a relatively young age.

Frontiers in psychology, 2018

Basic Emotion theory has had a tremendous influence on the affective sciences, including music psychology, where most researchers have assumed that music expressivity is constrained to a limited set of basic emotions. Several scholars suggested that these constrains to musical expressivity are explained by the existence of a shared acoustic code to the expression of emotions in music and speech prosody. In this article we advocate for a shift from this focus on basic emotions to a constructionist account. This approach proposes that the phenomenon of perception of emotions in music arises from the interaction of music's ability to express core affects and the influence of top-down and contextual information in the listener's mind. We start by reviewing the problems with the concept of Basic Emotions, and the inconsistent evidence that supports it. We also demonstrate how decades of developmental and cross-cultural research on music and emotional speech have failed to produce...

Scientific Study of Literature

This paper describes a structural account of phonetic symbolism and submits it to empirical investigation. To enable testing for possible iconic sound–emotion relations, participants compared pairs of syllables (e.g., ma – ba) as well as pairs of emotional states (e.g., joyful – sad) on various perceptual scales (e.g., softness). In addition, we replicated the classic ‘bouba/kiki’ experiment to investigate sound-shape symbolism. In accordance with the theoretical model, the results of the experimental tasks suggest that participants can detect abstract similarities between speech sounds and emotions as well as geometrical shapes. We discuss the theoretical model and the experimental results in relation to previous empirical findings and conflicting evidence from the study of affective iconicity in poetry.

Journal of Psycholinguistic Research, 2016

Previous research has demonstrated that even brief exposures to facial expressions of emotions elicit facial mimicry in receivers in the form of corresponding facial muscle movements. As well, vocal and verbal patterns of speakers converge in conversations, a type of vocal mimicry. There is also evidence of cross-modal mimicry in which emotional vocalizations elicit corresponding facial muscle activity. Further, empathic capacity has been associated with enhanced tendency towards facial mimicry as well as verbal synchrony. We investigated a type of potential cross-modal mimicry in a simulated dyadic situation. Specifically, we examined the influence of facial expressions of happy, sad, and neutral emotions on the vocal pitch of receivers, and its potential association with empathy. Results indicated that whereas both mean pitch and variability of pitch varied somewhat in the predicted directions, empathy was correlated with the difference in the variability of pitch while speaking to the sad and neutral faces. Discussion of results considers the dimensional nature of emotional vocalizations and possible future directions.

Empirical Studies of the Arts, 2021

Poems, as aesthetic objects, generate a subjective experience, which can be different for different readers. In this paper, we propose a method to quantify these subjective experiences. We gave participants three parallel excerpts and asked them to describe, in free text, the perceived emotive qualities of these excerpts. The descriptions were analysed quantitatively according to the dimensions of the Valence-Arousal-Dominance model of emotion. With the help of additional rating tasks and a structural theory of phonetic symbolism, we attribute the perceived emotive qualities to an interaction between the meaning of words, patterns of alliteration, and metric deviation.

Sensors

Over the past few years, virtual assistant solutions used in Contact Center systems are gaining popularity. One of the main tasks of the virtual assistant is to recognize the intentions of the customer. It is important to note that quite often the actual intention expressed in a conversation is also directly influenced by the emotions that accompany that conversation. Unfortunately, scientific literature has not identified what specific types of emotions in Contact Center applications are relevant to the activities they perform. Therefore, the main objective of this work was to develop an Emotion Classification for Machine Detection of Affect-Tinged Conversational Contents dedicated directly to the Contact Center industry. In the conducted study, Contact Center voice and text channels were considered, taking into account the following families of emotions: anger, fear, happiness, sadness vs. affective neutrality of the statements. The obtained results confirmed the usefulness of the...

Language and Speech, 2011

Prosody plays a variety of roles in infants’ communicative development, aiding in attention modulation, speech segmentation, and syntax acquisition. This study investigates the extent to which parents also spontaneously modulate prosodic aspects of infant directed speech in ways that distinguish semantic aspects of language. Fourteen mothers of two-year-old children read a picture book to their children in which they labeled pictures using dimensional adjectives (e.g., big, small, hot, cold). Recordings of the mothers’ input to their children were analyzed acoustically and antonyms within each dimension were compared. Mothers modulated aspects of their prosody including amplitude and duration of target words and sentences to distinguish dimensional adjectives. Mothers appear to recruit prosody in the service of word learning.

Palimpsestes, 2017

Revue de traduction 30 | 2017 Quand les sens font sens : traduire le "texte" filmique

Journal of Linguistics/Jazykovedný casopis, 2016

The paper tests the hypothesis that prosodic highlighting of individual words with emotional connotation can affect the recipient’s perception of the expressiveness of the entire utterance. Seventeen respondents listened to two speech realizations of the same text and marked its emotional level on the scale between positive and negative. To achieve emotional neutrality of the utterances, they were synthesized with Kempelen 3.0 speech system. The results show a tendency of respondents to perceive speech with prosodically highlighted negative words as more negative (sad) than speech with positive words highlighted, which was marked as more positive (joyful).

Behavior Research Methods, 2020

Automatic speech processing devices have become popular for quantifying amounts of ambient language input to children in their home environments. We assessed error rates for language input estimates for the Language ENvironment Analysis (LENA) audio processing system, asking whether error rates differed as a function of adult talkers' gender and whether they were speaking to children or adults. Audio was sampled from within LENA recordings from 23 families with children aged 4-34 months. Human coders identified vocalizations by adults and children, counted intelligible words, and determined whether adults' speech was addressed to children or adults. LENA's classification accuracy was assessed by parceling audio into 100-ms frames and comparing, for each frame, human and LENA classifications. LENA correctly classified adult speech 67% of the time across families (average false negative rate: 33%). LENA's adult word count showed a mean +47% error relative to human counts. Classification and Adult Word Count error rates were significantly affected by talkers' gender and whether speech was addressed to a child or an adult. The largest systematic errors occurred when adult females addressed children. Results show LENA's classifications and Adult Word Count entailed randomand sometimes largeerrors across recordings, as well as systematic errors as a function of talker gender and addressee. Due to systematic and sometimes high error in estimates of amount of adult language input, relying on this metric alone may lead to invalid clinical and/or research conclusions. Further validation studies and circumspect usage of LENA are warranted.

JMIR Mental Health, 2022

Background Emotions and mood are important for overall well-being. Therefore, the search for continuous, effortless emotion prediction methods is an important field of study. Mobile sensing provides a promising tool and can capture one of the most telling signs of emotion: language. Objective The aim of this study is to examine the separate and combined predictive value of mobile-sensed language data sources for detecting both momentary emotional experience as well as global individual differences in emotional traits and depression. Methods In a 2-week experience sampling method study, we collected self-reported emotion ratings and voice recordings 10 times a day, continuous keyboard activity, and trait depression severity. We correlated state and trait emotions and depression and language, distinguishing between speech content (spoken words), speech form (voice acoustics), writing content (written words), and writing form (typing dynamics). We also investigated how well these featu...

PLoS ONE, 2014

The Samburu pastoralists of Northern Kenya co-exist with African elephants, Loxodonta africana, and compete over resources such as watering holes. Audio playback experiments demonstrate that African elephants produce alarm calls in response to the voices of Samburu tribesmen. When exposed to adult male Samburu voices, listening elephants exhibited vigilance behavior, flight behavior, and produced vocalizations (rumbles, roars and trumpets). Rumble vocalizations were most common and were characterized by increased and more variable fundamental frequencies, and an upward shift in the first [F1] and second [F2] formant locations, compared to control rumbles. When exposed to a sequence of these recorded rumbles, roars and trumpets, listening elephants also exhibited vigilance and flight behavior. The same behavior was observed, in lesser degrees, both when the roars and trumpets were removed, and when the second formants were artificially lowered to levels typical of control rumbles. The ''Samburu alarm rumble'' is acoustically distinct from the previously described ''bee alarm rumble.'' The bee alarm rumbles exhibited increased F2, while Samburu alarm rumbles exhibited increased F1 and F2, compared to controls. Moreover, the behavioral reactions to the two threats were different. Elephants exhibited vigilance and flight behavior in response to Samburu and bee stimuli and to both alarm calls, but headshaking behavior only occurred in response to bee sounds and bee alarm calls. In general, increasingly threatening stimuli elicited alarm calls with increases in F 0 and in formant locations, and increasing numbers of these acoustic cues in vocal stimuli elicited increased vigilance and flight behavior in listening elephants. These results show that African elephant alarm calls differentiate between two types of threat and reflect the level of urgency of threats.

Frontiers in Psychology, 2015

Acoustic cues such as pitch height and timing are effective at communicating emotion in both music and speech. Numerous experiments altering musical passages have shown that higher and faster melodies generally sound "happier" than lower and slower melodies, findings consistent with corpus analyses of emotional speech. However, equivalent corpus analyses of complex time-varying cues in music are less common, due in part to the challenges of assembling an appropriate corpus. Here, we describe a novel, score-based exploration of the use of pitch height and timing in a set of "balanced" major and minor key compositions. Our analysis included all 24 Preludes and 24 Fugues from Bach's Well-Tempered Clavier (book 1), as well as all 24 of Chopin's Preludes for piano. These three sets are balanced with respect to both modality (major/minor) and key chroma ("A," "B," "C," etc.). Consistent with predictions derived from speech, we found major-key (nominally "happy") pieces to be two semitones higher in pitch height and 29% faster than minor-key (nominally "sad") pieces. This demonstrates that our balanced corpus of major and minor key pieces uses low-level acoustic cues for emotion in a manner consistent with speech. A series of post hoc analyses illustrate interesting trade-offs, with sets featuring greater emphasis on timing distinctions between modalities exhibiting the least pitch distinction, and vice-versa. We discuss these findings in the broader context of speech-music research, as well as recent scholarship exploring the historical evolution of cue use in Western music.

Current Sexual Health Reports, 2018

Purpose of the Review Rodents produce ultrasonic vocalizations (USV) under different social contexts, including courtship and reproduction. The present review aims to summarize the behavioral, bioacoustical, and physiological evidence that USV are reliable signals of sexual motivation in both male and female rodents. Recent Findings USV are actively produced by both sexes during sexual interactions, contrary to earlier assumptions. Maletypical and female-typical vocal behaviors can be identified. Calling rates and acoustic parameters, such as call duration, frequency, and energy, can be modulated rapidly over time by motivational state and sexual context. USV produced in response to sexual context could be regulated by the brain on a moment-to-moment basis through non-classical mechanisms of steroid action. Finally, I provide some practical considerations for the acoustic and statistical analyses of these vocal signals. Summary USV can be used as signals of sexual motivation in both sexes to study brain and hormonal mechanisms underlying sexual behavior or sexual differentiation.

Social Neuroscience, 2018

The ability to mentalize, or theory of mind (ToM), is sexually dimorphic in humans and impaired in schizophrenia. This sex-stratified study probed cognitive (indexed by intelligence) and affective (indexed by olfactory tasks) contributions to ToM performance in 37 individuals with schizophrenia and 31 healthy controls. The schizophrenia group showed impairments in mental state identification and inferring intentions compared to controls. Higher intelligence was correlated with mental state identification and inferring intentions in healthy females, whereas better smell identification was associated with mental state identification in healthy males. Conversely, higher intelligence was associated with mental state identification and inferring intentions in schizophrenia males, while better smell identification was correlated with mental state identification in schizophrenia females. These findings suggest that for ToM circuitry, the cognitive influences in healthy females and affective influences in healthy males are reversed in schizophrenia and may be displaced to lower circuitries by disease pathology. Symptom associations with emotion and cognition are also dimorphic, plausibly due to similar pathology superimposed on normal sex-specific circuitries. Males appear to rely on limbic processing for ToM, and disruption to this circuitry may contribute to development of negative symptoms. These findings highlight the importance of utilizing sex-stratified designs in schizophrenia research.