Papers by Marieke Hoetjes
Proceedings of the 11th International Conference on Speech Prosody, 2022
The present study aimed to disentangle the influence of gesture type, physical involvement level,... more The present study aimed to disentangle the influence of gesture type, physical involvement level, and individual differences in learner characteristics, i.e., working memory (WM) capacity and musicality, in determining the effectiveness of L2 lexical stress training. To this end, 60 native speakers of Dutch read aloud Spanish phrases containing cognates, which were counterbalanced for lexical stress position compared to their Dutch counterpart (e.g., 'piRÁmides' in Spanish, 'piraMIdes' in Dutch). They did so as a pre-test before receiving lexical stress training (T1) and as a post-test both directly after training (T2), and approximately one hour later (T3). Subjects received lexical stress training in one of five conditions varying in gesture type and physical involvement level: audiovisual (AV), AV-beat-perception, AV-beat-production, AVmetaphoric-perception, AV-metaphoric-production. Between T2 and T3, subjects performed a WM capacity and musical aptitude task. The results show that irrespective of training condition subjects significantly improved their L2 lexical stress production from T1 to T2 and T3. Although differences between training conditions were non-significant, there were several significant three-way interactions between WM capacity or musical aptitude and testing time and training condition. This underlines the importance of considering task and learner characteristics in determining the gestural benefit in learning L2 prosody.
Frontiers in Psychology, 2020
Most language learners have difficulties acquiring the phonemes of a second language (L2). Unfort... more Most language learners have difficulties acquiring the phonemes of a second language (L2). Unfortunately, they are often judged on their L2 pronunciation, and segmental inaccuracies contribute to miscommunication. Therefore, we aim to determine how to facilitate phoneme acquisition. Given the close relationship between speech and co-speech gesture, previous work unsurprisingly reports that gestures can benefit language acquisition, e.g., in (L2) word learning. However, gesture studies on L2 phoneme acquisition present contradictory results, implying that both specific properties of gestures and phonemes used in training, and their combination, may be relevant. We investigated the effect of phoneme and gesture complexity on L2 phoneme acquisition. In a production study, Dutch natives received instruction on the pronunciation of two Spanish phonemes, /u/ and /θ/. Both are typically difficult to produce for Dutch natives because their orthographic representation differs between both languages. Moreover, /θ/ is considered more complex than /u/, since the Dutch phoneme inventory contains /u/ but not /θ/. The instruction participants received contained Spanish examples presented either via audio-only, audiovisually without gesture, audiovisually with a simple, pointing gesture, or audiovisually with a more complex, iconic gesture representing the relevant speech articulator(s). Preceding and following training, participants read aloud Spanish sentences containing the target phonemes. In a perception study, Spanish natives rated the target words from the production study on accentedness and comprehensibility. Our results show that combining gesture and speech in L2 phoneme training can lead to significant improvement in L2 phoneme production, but both gesture and phoneme complexity affect successful learning: Significant learning only occurred for the less complex phoneme /u/ after seeing the more complex iconic gesture, whereas for the more complex phoneme /θ/, seeing the more complex gesture actually hindered acquisition. The perception results confirm the production findings and show that items containing /θ/ produced after receiving training with a less complex pointing gesture are considered less foreign-accented and more easily comprehensible as compared to the same items after audio-only training. This shows that gesture can facilitate task performance in L2 phonology acquisition, yet complexity affects whether certain gestures work better for certain phonemes than others.
Proceedings of the Gesture and Speech in Interaction conference (GeSpIn), 2019
This paper aims to study whether training with gestures benefits L2 phoneme acquisition from both... more This paper aims to study whether training with gestures benefits L2 phoneme acquisition from both a production and perception perspective. In the production study, Dutch learners of Spanish received pronunciation training for the phonemes /u/ and /θ/ in one of four conditions: audio-only, audiovisual , audiovisual with pointing gestures, or audiovisual with iconic gestures. Results show that in general, gestural training benefits L2 phoneme acquisition, but different gestures benefit the acquisition of different phonemes, possibly depending on their complexity. The perception study, in which L1 speakers of Spanish judged the L2 Spanish material on accentedness and comprehensibility, corroborate the findings from the production study: Including visual information in training generally lowered the perceived accentedness and increased the perceived comprehensibility of speech, but the type of phoneme matters. Together, these studies suggest that gestural training can benefit L2 phoneme acquisition, yet certain gestures work better for certain phonemes than others.
Proceedings of the 15th International Conference on Auditory-Visual Speech Processing (AVSP2019), 2019
The close relationship between speech and gesture has led to a range of studies focusing on the r... more The close relationship between speech and gesture has led to a range of studies focusing on the role of gesture in L1 and L2 language acquisition. However, few studies focused specifically on the acquisition of prosodic aspects of speech, and those that did, did not compare the possible effect of different types of gestures. Thus, this paper aims to determine whether seeing beat or metaphoric gestures during training facilitates L2 lexical stress acquisition as compared to seeing no gestures during training. Dutch participants received Spanish lexical stress training in one of three multimodal conditions and produced sentences containing cognates with diverging stress distribution in Dutch and Spanish before and afterwards. The results appear in line with our predictions in that learning proportions are higher after gestural training than after training without gestures and again higher after training with metaphoric gestures compared to training with beat gestures. However, a multinomial regression analysis reveals that only the presence vs. absence of a written accent in the cognate significantly affects L2 lexical stress acquisition. Hence, several factors are proposed that might explain our results and serve as a basis for future research .
Proceedings of the International Congress of Phonetic Sciences, 2019
Given that co-speech gestures affect language perception in both the L1 and L2, this paper aims t... more Given that co-speech gestures affect language perception in both the L1 and L2, this paper aims to determine whether they can also lead to improved L2 production. To this end, 51 native speakers of Dutch received training focused on the target-like pronunciation of the Spanish phonemes /θ/ and /u/, which are typically difficult to acquire for native speakers of Dutch. Participants were allocated to one of four training conditions: audio-only, audiovisual , audiovisual with pointing gestures, or audiovisual with iconic gestures. Before and after training, participants read aloud Spanish sentences that included words with /θ/ and /u/. Acoustic analysis revealed that /u/ is easier to acquire than /θ/ and that training modality affects on-target production. More specifically, all training conditions that included the visual modality lead to more on-target productions than the audio-only training. Interestingly, the effectiveness of the different types of multimodal training varies between the two phonemes.
CogSci Proceedings, July 2018 , 2018
There are cross-linguistics differences in the type of verb used to describe placement events. Du... more There are cross-linguistics differences in the type of verb used to describe placement events. Dutch uses semantically specific placement verbs (zetten, leggen), whereas English uses a semantically general placement verb (put). This semantic focus is reflected in speaker's gestures, which can be specific and object-focused by showing object-incorporating handshapes, or not. This study investigates the semantic placement event focus of Dutch L2 speakers of English, by investigating verb use and gesture production in placement event descriptions. Results showed that placement verb production was native-like, with a majority correct usage of put. However, gesture production showed many object-incorporating handshapes, similar to L1 Dutch gesture production. These results suggest that although the Dutch L2 speakers of English sounded native-like in speech, they were still trying to express Dutch-like placement verb meaning, by showing a continued focus on the object, as expressed in their gesture production.
Proceedings of the 4th European and 7th Nordic Symposium on Multimodal Communication (MMSYM 2016), Copenhagen, 29-30 September 2016, 2017
When objects or people are described repeatedly in conversation, the repeated references tend to ... more When objects or people are described repeatedly in conversation, the repeated references tend to be reduced variants of initial references. This can be seen both in speech, and in gesture. Previous studies focused on successful repeated references, produced in contexts of common ground. A question is whether repeated references are also reduced in contexts where there is less, or no common ground, for example during communicative problems. In particular, the present study asks whether gestures which are produced in repeated references following negative feedback become more informative for the addressee. Participants viewed silent video clips, each showing one gesture, taken either from object descriptions before any feedback was given, or from object descriptions given after (repeated) negative feedback. With each video clip participants were shown two objects. The task was to decide which of the two objects was the target associated with the gesture they were shown. Results showed that participants were better at this task when presented with gestures produced following (repeated) negative feedback. This leads us to conclude, firstly, that after having received negative feedback, gestures are not reduced, but become more informative, and secondly, that this might be done with the addressee in mind.
Journal of Multimodal Communication Studies, Special Issue: Gesture and Speech in Interaction, 2017
It has been hypothesized that speech and gesture work together, with people relying more on gestu... more It has been hypothesized that speech and gesture work together, with people relying more on gesture when speaking is hard, and relying more on speech when gesturing is hard. Indeed, previous work showed that when speaking is hard, gesture production can reduce cognitive load and thereby help the speech process. However, it is yet unknown what happens in speech and gesture production when gesturing is hard. In the current study, participants described complex tangram figures. Difficulty in either speech or gesture production was manipulated by having speakers do a secondary task which placed them under either verbal or motoric cognitive load. Results showed an increase in representational gesture rate when participants were under motoric load, as compared to the baseline condition, but only a weak effect of verbal load, and no difference between the verbal and motoric load conditions. Based on these findings, we conclude that making speaking hard caused a marginal increase in gesture production, and making gesturing hard actually led to more gestures. In sum, we find no evidence to support a two-way trade-off between speech and gestures. To our knowledge, this is the first study assessing the effects of a secondary motoric task on gesture production.
Proceedings of INTERSPEECH, 2017
Speech perception is multimodal, with not only speech, but also gesture presumably playing a role... more Speech perception is multimodal, with not only speech, but also gesture presumably playing a role in how a message is perceived. However, there have not been many studies on the effect that hand gestures may have on speech perception in general, and on persuasive speech in particular. Moreover, we do not yet know whether an effect of gestures may be larger when addressees are not involved in the topic of the discourse, and are therefore more focused on peripheral cues, rather than the content of the message. In the current study participants were shown a speech with or without gestures. Some participants were involved in the topic of the speech, others were not. We studied five measures of persuasiveness. Results showed that for all but one measure, viewing the video with accompanying gestures made the speech more persuasive. In addition, there were several interactions, showing that the performance of the speaker and the factual accuracy of the speech scored high especially for those participants who not only saw gestures but were also not involved in the topic of the speech.
The linguistic metaphors of time appear to influence how people gesture about time. This study fi... more The linguistic metaphors of time appear to influence how people gesture about time. This study finds that Chinese English bilinguals produce more vertical gestures when talking about Chinese time references with vertical spatial metaphors than (1) when talking about time conceptions in the English translations, and (2) when talking about Chinese time references with no spatial metaphors. Additionally, Chinese English bilinguals prefer vertical gestures to lateral gestures when perceiving Chinese time references with vertical spatial metaphors and the corresponding English translations, whereas there is no such preference when perceiving time references without spatial metaphors. Furthermore, this vertical tendency is not due to the fact that vertical gestures are generally less ambiguous than lateral gestures for addressees. In conclusion, the vertical gesturing about time by Chinese English bilinguals is shaped by both the stable language-specific conceptualisations, and the online changes in linguistic choices.
Previous studies found that repeated references in successful communication are often reduced, no... more Previous studies found that repeated references in successful communication are often reduced, not only at the acoustic level, but also in terms of words and manual co-speech gestures. In the present study, we investigated whether repeated references are still reduced in a situation when reduction would not be beneficial for the communicative situation, namely after the speaker receives negative feedback from the addressee. In a director–matcher task (experiment I), we studied gesture rate, as well as the general form of the gestures produced in initial and repeated references. In a separate experiment (experiment II) we studied whether there might (also) be more gradual differences in gesture form between gestures in initial and repeated references, by asking human judges which of two gestures (one from an initial and one from a repeated reference following negative feedback) they considered more precise. In both experiments, mutual visibility was added as a between subjects factor. Results showed that after negative feedback, gesture rate increased in a marginally significant way. With regard to gesture form, we found little evidence for changes in gesture form after negative feedback, except for a marginally significant increase of the number of repeated strokes within a gesture. Lack of mutual visibility only had a significant reducing effect on gesture size, and did not interact with repetition in any way. However, we did find gradual differences in gesture form, with gestures produced after negative feedback being judged as marginally more precise than initial gestures. The results from the present study suggest that in the production of unsuccessful repeated references, a process different from the reduction process as found in previous studies in repeated references takes place, with speakers appearing to put more effort into their gestures after negative feedback, as suggested by the data trending towards an increased gesture rate and towards gestures being judged as more precise after feedback.
In dialogue, repeated references contain fewer words (which are also acoustically reduced) and fe... more In dialogue, repeated references contain fewer words (which are also acoustically reduced) and fewer gestures than initial ones. In this paper, we describe three experiments studying to what extent gesture reduction is comparable to other forms of linguistic reduction. Since previous studies showed conflicting findings for gesture rate, we systematically compare two measures of gesture rate: gesture rate per word and per semantic attribute (Experiment I). In addition, we ask whether repetition impacts the form of gestures, by manual annotation of a number of features (Experiment I), by studying gradient differences using a judgment test (Experiment II), and by investigating how effective initial and repeated gestures are at communicating information (Experiment III). The results revealed no reduction in terms of gesture rate per word, but a U-shaped reduction pattern for gesture rate per attribute. Gesture annotation showed no reliable effects of repetition on gesture form, yet participants judged gestures from repeated references as less precise than those from initial ones. Despite this gradient reduction, gestures from initial and repeated references were equally successful in communicating information. Besides effects of repetition, we found systematic effects of visibility on gesture production, with more, longer, larger and more communicative gestures when participants could see each other. We discuss the implications of our findings for gesture research and for models of speech and gesture production.
Do people speak differently when they cannot use their hands? Previous studies have suggested tha... more Do people speak differently when they cannot use their hands? Previous studies have suggested that speech becomes less fluent and more monotonous when speakers cannot gesture, but the evidence for this claim remains inconclusive. The present study attempts to find support for this claim in a production experiment in which speakers had to give addressees instructions on how to tie a tie; half of the participants had to perform this task while sitting on their hands. Other factors that influence the ease of communication, such as mutual visibility and previous experience, were also taken into account. No evidence was found for the claim that the inability to gesture affects speech fluency or monotony. An additional perception task showed that people were also not able to hear whether someone gestures or not.
Previous research on speech and gesture has found that repeated references are often linguistical... more Previous research on speech and gesture has found that repeated references are often linguistically reduced in terms of, for example, the number of words and the acoustic realization of these words, compared to initial references. The present study looks at the production of repeated references by 14 signers of Sign Language of the Netherlands (NGT). Participants had to describe figures to an addressee, who had to pick the correct figure from a large group of figures. Several figures had to be described several times. The question was whether there would be reduction in the repeated references. We found systematic effects of repetition in that repeated references were shorter, contained fewer signs, and shorter signs than initial references. Moreover, in order to measure sign precision, a perception test was used where participants had to judge, in a forced choice task, which sign they considered to be the most precise, looking at 40 pairs of video clips with signs produced in either initial or repeated references to the same object by the same signer. We found that, non-signing participants (but not signing participants) consider signs produced during repeated references to be less precise than the signs produced during initial references. Taking together these results suggest that a similar reduction process in repeated references occurs in NGT as has been found previously for speech and gesture.
Proceedings of the 36th Annual Conference of the Cognitive Science Society, 2014
Does language influence the production and perception of gestures? The metaphorical use of langua... more Does language influence the production and perception of gestures? The metaphorical use of language in representing time is deeply interlinked with actions in space, such as gestures. In Chinese, speakers can talk and gesture about time as if it were horizontal, sagittal, or vertical. In English, speakers rarely employ the vertical plane. Two experiments showed that the verbal use of vertical spatial metaphors had an online influence on the production and perception of gestures by late Chinese-English bilinguals. Participants produced more vertical gestures when talking about time references by use of vertical spatial metaphors, e.g. 'shàngzhōu' (literally: 'above week', meaning 'last week'), and they preferred vertical gestures to horizontal gestures when perceiving time references with vertical spatial metaphors. Gestures are not only shaped by the language specific conceptualisation, but are also sensitive to the changes in linguistic choices, both in production and perception.
Uploads
Papers by Marieke Hoetjes