The NOMCO multimodal Nordic resource–goals and characteristics

2011

Proceedings of the 3rd Nordic Symposium on Multimodal Communication. Editors: Patrizia Paggio, Elisabeth Ahlsen, Jens Allwood, Kristiina Jokinen, Costanza Navarretta. NEALT Proceedings Series, Vol. 15 (2011), 72–77. © 2011 The editors and contributors. Published by Northern European Association for Language Technology (NEALT) http://omilia.uio.no/nealt . Electronically published at Tartu University Library (Estonia) http://hdl.handle.net/10062/22532 .

Italian Journal of Computational Linguistics, 2021

Modeling dialogue implies detecting natural interaction. A pragmatic approach allows to consider the linguistic act composed of several and different features interacting with each other. Data collected for this project comprises three different genres of communication: monological, dialogical and conversational. The project aims to identify and analyze the pragmatic value of multimodal communication spotting the linguistic actions which carry out illocution values. We draw a pragmatic approach to study multimodal interaction combining the L-AcT annotation (Cresti 2000) with the gesture's architecture designed by Kendon (Kendon 2004). The annotation system is designed to divide the speech units (utterance, intonation units and illocution types) (Hart, Collier, and Cohen 2006) (Cresti 2005) (Moneglia and Raso 2014) from gestural units (Gesture Unit, Gesture Phrase, Gesture Phase). Keeping the Gesture Unit as a superior macrounit at the other gestural units only for the quantitative purpose, we realize a matching between gesture and speech units. These units work together to form the communicative intention of the speaker that can be recognizable by the Illocution Type. This annotation system leads to understanding how speakers realize multimodal linguistic actions and how different modalities work.

The Routledge Handbook of English Language and Digital Humanities, 2020

In the age of the Internet, trillions of bytes of media data are generated every day through telecommunications and social media. This surge of born-digital media data, for example, instant voice/video messages, conference calls, podcasts, video blogs and so on, offers researchers unprecedented opportunities to deepen their understanding of how human beings communicate and go about their social activities. However, such a large amount of data also brings a new problem: how may we plough through so much media data and extract meaningful information efficiently? This chapter explores opportunities and challenges at the interface between digital humanities and multimodality research which focuses on the use of prosody and gesture in spoken communication. Following an overview of key methods and frameworks in prosody and gestures research, it highlights selected projects which have showcased the ways in which today’s computer technology has revolutionised multimodality as an area of research. In recent years, many new computer tools have become available to aid media data acquisition, processing and analysis. These tools have (semi-)automatised many processes which were labour-intensive, expensive and tedious. Therefore, researchers can now afford to compile and process substantially larger multimodal datasets much faster and at a much lower cost. The chapter also introduces tools which open up new avenues for researchers to acquire new types of multimodal data (e.g. YouTube videos) and data streams (e.g. GPS, heartbeats). In the sample analysis, we demonstrate the typical workflow for using a range of these latest computer tools to generate a corpus of YouTube videos, automatically annotate prosodic patterns, align multiple data streams and perform a multimodal analysis on the use of the epistemic stance marker ‘I think’ in video blogs.

2011

This paper deals with the way in which feedback is expressed through speech and gestures in the Danish NOMCO corpus of dyadic first encounters. The annotation includes the speech transcription as well as attributes concerning shape and conversational function of head movements and facial expressions. Our analysis of the data shows that all communication modalities, i.e. head, face and eyebrows, contribute to the expressions of feedback, with repeated nods and smiles as the most frequent feedback gesture types. In general, the use of nods as feedback gestures in our data is comparable to what earlier studies have found for other languages, but feedback is also often expressed by other head movements and smiles.

2012

This paper focusses on multimodal activity and its functions, especially as a communicative means to structure the discourse among the interlocutors: to give feedback and indicate turn-takings and mutual agreement. Starting from the assumption that natural language communication is a holistic process which aims at creating shared understanding, and requires interpretation of vocal and visual signals as part of successful interaction, the paper aims to form a coherent picture of the participants’ multimodal communication strategies and their engagement in the conversation. It presents observations on the conversational feedback and turn-taking functions especially related to head movement, hand gesturing, and body posture. The main claim concerns the meta-discursive function of visual signals, related to their use as unobtrusive means to control the interaction and to construct shared understanding. The paper deals with synchrony between head movements, hand gesturing and body postur...

ACM Transactions on Computer-Human Interaction, 2002

Gesture and speech combine to form a rich basis for human conversational interaction. To exploit these modalities in HCI, we need to understand the interplay between them and the way in which they support communication. We propose a framework for the gesture research done to date, and present our work on the cross-modal cues for discourse segmentation in free-form gesticulation accompanying speech in natural conversation as a new paradigm for such multimodal interaction. The basis for this integration is the psycholinguistic concept of the coequal generation of gesture and speech from the same semantic intent. We present a detailed case study of a gesture and speech elicitation experiment in which a subject describes her living space to an interlocutor. We perform F. Quek et al.