Feedback in Nordic First-Encounters: a Comparative Study

2003

GPTL Gothenburg Papers in Theoretical Linguistics Papers in GPTL cover general and theoretical topics in linguistics. The series contain papers from single individuals and project groups. It appears irregularly in two subseries, blue for English Papers, and green for Swedish Papers. JENS ALLWOOD Editor, GPTL ... Contents Þórunn Blöndal Feedback in Conversational Storytelling................... ........... 1–17 Loredana Cerrato On the Acoustic, Prosodic and Gestural Characteristics of “m-like” Sounds in Swedish............................................ 18–31 Nina Sværke Hansen ...

2007

This thesis deals with human communicative behaviour related to feedback, analysed across languages (Italian and Swedish), modalities (auditory versus visual) and different communicative situations (human-human versus human-machine dialogues). The aim of this study is to give more insight into how humans use communicative behaviour related to feedback and at the same time to suggest a method to collect valuable data that can be useful to control facial and head movements related to visual feedback in synthetic conversational agents. The study of human communicative behaviour necessitates the good quality of the materials under analysis, the support of reliable software packages for the audio-visual analysis and a specific coding scheme for the annotation of the phenomena under observation. The materials used for the investigations presented in this thesis span from spontaneous conversations video recorded in real communicative situations, and semi-spontaneous dialogues obtained with different eliciting techniques, such as map-task and information-seeking scenarios, to a specific corpus of controlled interactive speech collected by means of a motion capture system. When motion caption is used it is possible to register facial and head movements with a high degree of precision, so as to obtain valuable data useful for the implementation of facial displays in talking heads. A specific coding scheme has been developed, tested and used to annotate feedback. The annotation has been carried out with the support of different available software packages for audio-visual analysis. The procedure followed in this thesis involves initial analyses of communicative phenomena in spontaneous human-human dialogues and human-machine interaction, in order to learn about regularities in human communicative behaviour that could be transferred to talking heads, then, for the sake of reproduction in talking heads, the investigation includes more detailed analyses of data collected in a lab environment with a novel acquisition set-up that allows capturing the dynamics of facial and head movements. Finally the possibilities of transferring human communicative behaviour to a talking face are discussed and some evaluation paradigms are illustrated. The idea of reproducing human behaviour in talking heads is based on the assumption that the reproduction of facial displays related to communicative phenomena such as turn management, feedback production and expression of emotions in embodied conversational agents, might result in the design of advanced systems capable of effective multi-modal interactions with humans.

rapport nr.: Gothenburg Papers in Theoretical Linguistics 91, 2005

GPTL Gothenburg Papers in Theoretical Linguistics Papers in GPTL cover general and theoretical topics in linguistics. The series contain papers from single individuals and project groups. It appears irregularly in two subseries, blue for English Papers, and green for Swedish Papers. JENS ALLWOOD Editor, GPTL ... Contents Þórunn Blöndal Feedback in Conversational Storytelling................... ........... 1–17 Loredana Cerrato On the Acoustic, Prosodic and Gestural Characteristics of “m-like” Sounds in Swedish............................................ 18–31 Nina Sværke Hansen ...

There is wide agreement about the fact that different kinds of communicative gestures like head movements and facial expressions are used extensively in face-to-face communication for various purposes e.g. expressing feedback, emphasis, turn management and emotions, or supporting the speaker's own communication management. Moreover, studies show that individual differences in gestural behaviour can be attributed to different personality traits like extraversion and openness. But these findings do not model the fact that the same person may feel and act differently in different situations. We compare the results of postexperiment questionnaires consisting of self-rating questions about conversation experience, influence and attitudinal emotions, to gestural behaviour in a multimodal corpus of face-to-face conversations. Our findings suggest a slight positive correlation between the way conversations are perceived and the number of head nods produced. In turn, the degree of head movement in the specific conversation is probably a function of the subject's positive engagement in it.

In this paper we examine the relationship between specific head movement events – head nods, often treated as prosodic boundary markers – and syntactic units in Finnish (FinSL) and Swedish Sign Language (SSL). In the study we investigated the alignment of head nods with syntactic units on the basis of a total of 20 (10+10) FinSL and SSL narratives. The results of the study show that in both languages head nods appeared similarly on syntactic boundaries and that the tendency was to align nods sentence-finally. However, not all head nods behaved this way: for example, a relatively large number of head nods were also found to occur sentence-initially or elsewhere in the sentence. Furthermore, head nods occurring on syntactic boundaries also had non-boundary marking functions, and not all syntactic boundaries occurred with head nods.

2016

This paper presents an analysis of the temporal alignment between head movements and associated speech segments in the NOMCO corpus of first encounter dialogues [1]. Our results show that head movements tend to start slightly before the onset of the corresponding speech sequence and to end slightly after, but also that there are delays in both directions in the range of -/+ 1s. Various factors that may influence delay duration are investigated. Correlations are found between delay length and the duration of the speech sequences associated with the head movements. Effects due to the different head movement types are also discussed.

2023

We present first results of a study investigating the salience and typicality of prosodic markers in Swedish at turn ends for turnyielding and turn-keeping purposes. We performed an experiment where participants (N=32) were presented with conversational chunks and, after the audio ended, were asked to determine which of two speakers would speak next by clicking a picture on a screen. Audio stimuli were manipulated by (i) raising and (ii) lowering f 0 over the last 500 ms of a turn, (iii) speeding up or (iv) slowing down duration over the last 500 ms, and (v) raising and (vi) lowering the last pitch peak. In our data, out of all manipulations, increasing the speech rate was found to be the most disruptive (p<.005). Higher speech rate led to longer reaction times in turn-keeping, which were shorter in turn-yielding. Other manipulations did not significantly alter reaction times. The results presented here may be complemented with eye movement data, to further elucidate cognitive mechanisms underlying turn-taking behavior.

Speech Prosody 2016, 2016

In this paper we examine the relationship between specific head movement events-head nods, often treated as prosodic boundary markers-and syntactic units in Finnish (FinSL) and Swedish Sign Language (SSL). In the study we investigated the alignment of head nods with syntactic units on the basis of a total of 20 (10+10) FinSL and SSL narratives. The results of the study show that in both languages head nods appeared similarly on syntactic boundaries and that the tendency was to align nods sentence-finally. However, not all head nods behaved this way: for example, a relatively large number of head nods were also found to occur sentence-initially or elsewhere in the sentence. Furthermore, head nods occurring on syntactic boundaries also had non-boundary marking functions, and not all syntactic boundaries occurred with head nods.

Language Resources and Evaluation, 2016

This article presents the Danish NOMCO Corpus, an annotated multimodal collection of video-recorded first acquaintance conversations between Danish speakers. The annotation includes speech transcription including word boundaries, and formal as well as functional coding of gestural behaviours, specifically head movements, facial expressions, and body posture. The corpus has served as the empirical basis for a number of studies of communication phenomena related to turn management, feedback exchange, information packaging and the expression of emotional attitudes. We describe the annotation scheme, procedure, and annotation results. We then summarise a number of studies conducted on the corpus. The corpus is available for research and teaching purposes through the authors of this article.