Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Materials and Methods
Participants
Design
Materials
Procedure
Data Analysis
Data Availability
Use of Human Participants
Results
Discussion
Conclusions
References
All Topics
Psychology

Language Experience Changes Audiovisual Perception

Profile image of Tuan LamTuan Lam

2018, Brain sciences

https://doi.org/10.3390/BRAINSCI8050085
visibility

…

description

14 pages

link

1 file

Abstract

Can experience change perception? Here, we examine whether language experience shapes the way individuals process auditory and visual information. We used the McGurk effect—the discovery that when people hear a speech sound (e.g., “ba”) and see a conflicting lip movement (e.g., “ga”), they recognize it as a completely new sound (e.g., “da”). This finding suggests that the brain fuses input across auditory and visual modalities demonstrates that what we hear is profoundly influenced by what we see. We find that cross-modal integration is affected by language background, with bilinguals experiencing the McGurk effect more than monolinguals. This increased reliance on the visual channel is not due to decreased language proficiency, as the effect was observed even among highly proficient bilinguals. Instead, we propose that the challenges of learning and monitoring multiple languages have lasting consequences for how individuals process auditory...

Related papers

Impact of language on development of auditory-visual speech perception
Kaoru Sekiyama

Developmental Science, 2008

The McGurk effect paradigm was used to examine the developmental onset of inter-language differences between Japanese and English in auditory-visual speech perception. Participants were asked to identify syllables in audiovisual (with congruent or discrepant auditory and visual components), audio-only, and video-only presentations at various signal-to-noise levels. In Experiment 1 with two groups of adults, native speakers of Japanese and native speakers of English, the results on both percent visually influenced responses and reaction time supported previous reports of a weaker visual influence for Japanese participants. In Experiment 2, an additional three age groups (6, 8, and 11 years) in each language group were tested. The results showed that the degree of visual influence was low and equivalent for Japanese and English language 6-year-olds, and increased over age for English language participants, especially between 6 and 8 years, but remained the same for Japanese participants. This may be related to the fact that English language adults and older children processed visual speech information relatively faster than auditory information whereas no such inter-modal differences were found in the Japanese participants' reaction times.

View PDFchevron_right
Effects of language experience: neural commitment to language-specific auditory patterns
Patricia Kuhl

NeuroImage, 2005

Linguistic experience alters an individual's perception of speech. We here provide evidence of the effects of language experience at the neural level from two magnetoencephalography (MEG) studies that compare adult American and Japanese listeners' phonetic processing. The experimental stimuli were American English /ra/ and /la/ syllables, phonemic in English but not in Japanese. In Experiment 1, the control stimuli were /ba/ and /wa/ syllables, phonemic in both languages; in Experiment 2, they were non-speech replicas of /ra/ and /la/. The behavioral and neuromagnetic results showed that Japanese listeners were less sensitive to the phonemic /r-l/ difference than American listeners. Furthermore, processing non-native speech sounds recruited significantly greater brain resources in both hemispheres and required a significantly longer period of brain activation in two regions, the superior temporal area and the inferior parietal area. The control stimuli showed no significant differences except that the duration effect in the superior temporal cortex also applied to the non-speech replicas. We argue that early exposure to a particular language produces a bneural commitmentQ to the acoustic properties of that language and that this neural commitment interferes with foreign language processing, making it less efficient.

View PDFchevron_right
Brain activation during audiovisual exposure anticipates future perception of ambiguous speech
Elia Formisano, Jean Vroomen

NeuroImage, 2011

In modern perceptual neuroscience, the focus of interest has shifted from a restriction to individual modalities to an acknowledgement of the importance of multisensory processing. One particularly well-known example of cross-modal interaction is the McGurk illusion. It has been shown that this illusion can be modified, such that it creates an auditory perceptual bias that lasts beyond the duration of audiovisual stimulation, a process referred to as cross-modal recalibration . Recently, we have suggested that this perceptual bias is stored in auditory cortex, by demonstrating the feasibility of retrieving the subjective perceptual interpretation of recalibrated ambiguous phonemes from functional magnetic resonance imaging (fMRI) measurements in these regions . However, this does not explain which brain areas integrate the information from the two senses and represent the origin of the auditory perceptual bias. Here we analyzed fMRI data from audiovisual recalibration blocks, utilizing behavioral data from perceptual classifications of ambiguous auditory phonemes that followed these blocks later in time. Adhering to this logic, we could identify a network of brain areas (bilateral inferior parietal lobe [IPL], inferior frontal sulcus [IFS], and posterior middle temporal gyrus [MTG]), whose activation during audiovisual exposure anticipated auditory perceptual tendencies later in time. We propose a model in which a higher-order network, including IPL and IFS, accommodates audiovisual integrative learning processes, which are responsible for the installation of a perceptual bias in auditory regions. This bias then determines constructive perceptual processing.

View PDFchevron_right
Language Effects on the Degree of Visual Influence In Audiovisual Speech Perception
Valerie Hazan

Proceedings of the 16th International Congress of …, 2007

View PDFchevron_right
Inter-language differences in the McGurk effect for Dutch and Cantonese listeners
Jean Vroomen

1995

A group of Dutch and Cantonese listeners were compared on a audio-visual speech perception task. Using vid eo techniques, lipmovements of syllables were d ubbed on a speech signal such that the heard and seen place of articulation d id not match . The Cantonese participants were more influenced by vision than the Dutch. We suggest that the phonological repertoire has an influence on aud io-visual speech perception.

View PDFchevron_right
Cultural and linguistic factors in audiovisual speech processing: The McGurk effect in Chinese subjects
Kaoru Sekiyama

Perception & Psychophysics, 1997

View PDFchevron_right
Electrophysiological evidence for a multisensory speech-specific mode of perception
Jean Vroomen

Neuropsychologia, 2012

We investigated whether the interpretation of auditory stimuli as speech or non-speech affects audiovisual (AV) speech integration at the neural level. Perceptually ambiguous sine-wave replicas (SWS) of natural speech were presented to listeners who were either in 'speech mode' or 'non-speech mode'. At the behavioral level, incongruent lipread information led to an illusory change of the sound only for listeners in speech mode. The neural correlates of this illusory change were examined in an audiovisual mismatch negativity (MMN) paradigm with SWS sounds. In an oddball sequence, 'standards' consisted of SWS/onso/coupled with lipread/onso/, and 'deviants' consisted of SWS/onso/coupled with lipread/omso/. The AV deviant induced a McGurk-MMN for listeners in speech mode, but not for listeners in non-speech mode. These results demonstrate that the illusory change in the sound by incongruent lipread information evoked an MMN which presumably takes place at a pre-attentive sensory processing stage.

View PDFchevron_right
Brain activity during audiovisual speech perception: An fMRI study of the McGurk e¡ect
Daniel Callan

2003

fMRI was used to assess the relationship between brain activation and the degree of audiovisual integration of speech information during a phoneme categorization task. Twelve subjects heard a speaker say the syllable /aba/ paired either with video of the speaker saying the same consonant or a di¡erent one (/ava/). In order to manipulate the degree of audiovisual integration, the audio was either synchronous or 7 400 ms out of phase with the visual stimulus. Subjects reported whether they heard the consonant /b/ or another consonant; fewer /b/ responses when the audio and visual stimuli were mismatched indicated higher levels of visual in£uence on speech perception (McGurk e¡ect). Active brain regions during presentation of the incongruent stimuli included the superior temporal and inferior frontal gyrus, as well as extrastriate, premotor and posterior parietal cortex. A regression analysis related the strength of the McGurk e¡ect to levels of brain activation. Paradoxically, higher numbers of /b/ responses were positively correlated with activation in the left occipito-temporal junction, an area often associated with processing visual motion. This activation suggests that auditory information modulates visual processing to a¡ect perception.

View PDFchevron_right
Timing of auditory-visual integration in the McGurk effect
Ken Grant

Society of Neuroscience annual meeting, San Diego, CA, 2001

In the following set of experiments, we used the McGurk illusion first reported by McGurk and McDonald (1976) to examine multisensory integration. In its “fusion” component, the illusion emerges when a participant is presented with an auditory bilabial (eg/ba/) dubbed onto a visual velar (eg articulatory movement/ga/). Under these conditions participants consistently report hearing an alveolar/da/or/∆ a/, virtual percept resulting from the AV fusion.

View PDFchevron_right
Multisensory integration of speech sounds with letters vs. visual speech: only visual speech induces the mismatch negativity
Jean Vroomen

European Journal of Neuroscience, 2018

Numerous studies have demonstrated that the vision of lip movements can alter the perception of auditory speech syllables (McGurk effect). While there is ample evidence for integration of text and auditory speech, there are only a few studies on the orthographic equivalent of the McGurk effect. Here, we examined whether written text, like visual speech, can induce an illusory change in the perception of speech sounds on both the behavioural and neural levels. In a sound categorization task, we found that both text and visual speech changed the identity of speech sounds from an /aba/‐/ada/ continuum, but the size of this audiovisual effect was considerably smaller for text than visual speech. To examine at which level in the information processing hierarchy these multisensory interactions occur, we recorded electroencephalography in an audiovisual mismatch negativity (MMN, a component of the event‐related potential reflecting preattentive auditory change detection) paradigm in which ...

View PDFchevron_right

References (64)

  1. Marks, L.E. On colored-hearing synesthesia: Cross-modal translations of sensory dimensions. Psychol. Bull. 1975, 82, 303-331. [CrossRef] [PubMed]
  2. Ward, J.; Simner, J.; Auyeung, V. A comparison of lexical-gustatory and grapheme-colour synaesthesia. Cogn. Neuropsychol. 2005, 22, 28-41. [CrossRef] [PubMed]
  3. Burnham, D.; Dodd, B. Auditory-visual speech integration by prelinguistic infants: Perception of an emergent consonant in the McGurk effect. Dev. Psychobiol. 2004, 45, 204-220. [CrossRef] [PubMed]
  4. Maier, J.X.; Neuhoff, J.G.; Logothetis, N.K.; Ghazanfar, A.A. Multisensory integration of looming signals by rhesus monkeys. Neuron 2004, 43, 177-181. [CrossRef] [PubMed]
  5. Frank, R.A.; Byram, J. Taste-smell interactions are tastant and odorant dependent. Chem. Senses 1988, 13, 445-455. [CrossRef]
  6. Reinoso Carvalho, F.; Wang, Q.J.; van Ee, R.; Persoone, D.; Spence, C. "Smooth operator": Music modulates the perceived creaminess, sweetness, and bitterness of chocolate. Appetite 2017, 108, 383-390. [CrossRef] [PubMed]
  7. Ho, H.N.; Iwai, D.; Yoshikawa, Y.; Watanabe, J.; Nishida, S. Combining colour and temperature: A blue object is more likely to be judged as warm than a red object. Sci. Rep. 2014, 4, 5527. [CrossRef] [PubMed]
  8. Saldaña, H.M.; Rosenblum, L.D. Visual influences on auditory pluck and bow judgments. Percept. Psychophys. 1993, 54, 406-416. [CrossRef] [PubMed]
  9. Ross, L.A.; Saint-Amour, D.; Leavitt, V.M.; Javitt, D.C.; Foxe, J.J. Do You See What I Am Saying? Exploring Visual Enhancement of Speech Comprehension in Noisy Environments. Cereb. Cortex 2006, 17, 1147-1153. [CrossRef] [PubMed]
  10. Mcgurk, H.; Macdonald, J. Hearing lips and seeing voices. Nature 1976, 264, 746-748. [CrossRef] [PubMed]
  11. Grant, K.W.; Seitz, P.-F. The use of visible speech cues for improving auditory detection of spoken sentences. J. Acoust. Soc. Am. 2000, 108, 1197-1208. [CrossRef] [PubMed]
  12. Fixmer, E.; Hawkins, S. The influence of quality of information on the McGurk effect. In Proceedings of the AVSP'98 International Conference on Auditory-Visual Speech Processing, Terrigal, Australia, 5 December 1998; pp. 27-32.
  13. Sekiyama, K.; Kanno, I.; Miura, S.; Sugita, Y. Auditory-visual speech perception examined by fMRI and PET. Neurosci. Res. 2003, 47, 277-287. [CrossRef]
  14. Sekiyama, K.; Tohkura, Y. McGurk effect in non-English listeners: Few visual effects for Japanese subjects hearing Japanese syllables of high auditory intelligibility. J. Acoust. Soc. Am. 1991, 90, 1797-1805. [CrossRef] [PubMed]
  15. Chen, Y.; Hazan, V. Language Effects on the Degree of Visual Influence in Audiovisual Speech Perception. In Proceedings of the 16th International Congress of Phonetic Sciences, Saarbrueken, Germany, 6-10 August 2007; pp. 2177-2180.
  16. Sekiyama, K.; Tohkura, Y. Inter-language differences in the influence of visual cues in speech perception. J. Phonetics 1993, 21, 427-444.
  17. Sekiyama, K. Differences in auditory-visual speech perception between Japanese and Americans: McGurk effect as a function of incompatibility. J. Acoust. Soc. Jpn. 1994, 15, 143-158. [CrossRef]
  18. Wang, Y.; Behne, D.M.; Jiang, H. Influence of native language phonetic system on audio-visual speech perception. J. Phonetics 2009, 37, 344-356. [CrossRef]
  19. Bialystok, E. Bilingualism in Development; Cambridge University Press: Cambridge, UK, 2001; ISBN 9780511605963.
  20. Costa, A.; Hernández, M.; Sebastián-Gallés, N. Bilingualism aids conflict resolution: Evidence from the ANT task. Cognition 2008, 106, 59-86. [CrossRef] [PubMed]
  21. Cummins, J. Bilingualism and the development of metalinguistic awareness. J. Cross Cult. Psychol. 1978, 9, 131-149. [CrossRef]
  22. Galambos, S.J.; Goldin-Meadow, S. The effects of learning two languages on levels of metalinguistic awareness. Cognition 1990, 34, 1-56. [CrossRef]
  23. Bialystok, E.; Craik, F.I.M.; Freedman, M. Bilingualism as a protection against the onset of symptoms of dementia. Neuropsychologia 2007, 45, 459-464. [CrossRef] [PubMed]
  24. Alladi, S.; Bak, T.H.; Duggirala, V.; Surampudi, B.; Shailaja, M.; Shukla, A.K.; Chaudhuri, J.R.; Kaul, S. Bilingualism delays age at onset of dementia, independent of education and immigration status. Neurology 2013, 81, 1938-1944. [CrossRef] [PubMed]
  25. Goetz, P.J. The effects of bilingualism on theory of mind development. Biling. Lang. Cogn. 2003, 6, 1-15. [CrossRef]
  26. Fan, S.P.; Liberman, Z.; Keysar, B.; Kinzler, K.D. The exposure advantage: Early exposure to a multilingual environment promotes effective communication. Psychol. Sci. 2015, 26, 1090-1097. [CrossRef] [PubMed]
  27. Bradlow, A.R.; Bent, T. The clear speech effect for non-native listeners. J. Acoust. Soc. Am. 2002, 112, 272-284. [CrossRef] [PubMed]
  28. Mayo, L.H.; Florentine, M.; Buus, S. Age of second-language acquisition and perception of speech in noise. J. Speech Lang. Hear. Res. 1997, 40, 686-693. [CrossRef] [PubMed]
  29. Gollan, T.H.; Montoya, R.I.; Cera, C.; Sandoval, T.C. More use almost always means a smaller frequency effect: Aging, bilingualism, and the weaker links hypothesis. J. Mem. Lang. 2008, 58, 787-814. [CrossRef] [PubMed]
  30. Gollan, T.H.; Slattery, T.J.; Goldenberg, D.; Van Assche, E.; Duyck, W.; Rayner, K. Frequency drives lexical access in reading but not in speaking: The frequency-lag hypothesis. J. Exp. Psychol. Gen. 2011, 140, 186-209. [CrossRef] [PubMed]
  31. Marian, V.; Spivey, M. Competing activation in bilingual language processing: Within-and between-language competition. Biling. Lang. Cogn. 2018, 6, 97-115. [CrossRef]
  32. Thierry, G.; Wu, Y.J. Brain potentials reveal unconscious translation during foreign-language comprehension. Proc. Natl. Acad. Sci. USA 2007, 104, 12530-12535. [CrossRef] [PubMed]
  33. Best, C.T. The emergence of native-language phonological influences in infants: A perceptual assimilation model. In The Development of Speech Perception: The Transition from Speech Sounds to Spoken Words; The MIT Press: Cambridge, MA, USA, 1994; Volume 167, pp. 167-224.
  34. Kuhl, P.; Williams, K.; Lacerda, F.; Stevens, K.; Lindblom, B. Linguistic experience alters phonetic perception in infants by 6 months of age. Science 1992, 121-154. [CrossRef]
  35. Sebastián-Gallés, N.; Soto-Faraco, S. Online processing of native and non-native phonemic contrasts in early bilinguals. Cognition 1999, 72, 111-123. [CrossRef]
  36. Fennell, C.T.; Byers-Heinlein, K.; Werker, J.F. Using speech sounds to guide word learning: The case of bilingual infants. Child Dev. 2007, 78, 1510-1525. [CrossRef] [PubMed]
  37. Navarra, J.; Soto-Faraco, S. Hearing lips in a second language: Visual articulatory information enables the perception of second language sounds. Psychol. Res. 2007, 71, 4-12. [CrossRef] [PubMed]
  38. Pons, F.; Bosch, L.; Lewkowicz, D.J. Bilingualism modulates infants' selective attention to the mouth of a talking face. Psychol. Sci. 2015, 26, 490-498. [CrossRef] [PubMed]
  39. Marian, V.; Blumenfeld, H.K.; Kaushanskaya, M. The language experience and proficiency questionnaire (LEAP-Q): Assessing language profiles in bilinguals and multilinguals. J. Speech Lang. Hear. Res. 2007, 50, 940-967. [CrossRef]
  40. Van Wassenhove, V.; Grant, K.W.; Poeppel, D. Visual speech speeds up the neural processing of auditory speech. Proc. Natl. Acad. Sci. USA 2005, 102, 1181-1186. [CrossRef] [PubMed]
  41. Van Wassenhove, V.; Grant, K.W.; Poeppel, D. Temporal window of integration in auditory-visual speech perception. Neuropsychologia 2007, 45, 598-607. [CrossRef] [PubMed]
  42. Alsius, A.; Navarra, J.; Campbell, R.; Soto-Faraco, S. Audiovisual integration of speech falters under high attention demands. Curr. Biol. 2005, 15, 839-843. [CrossRef] [PubMed]
  43. Bialystok, E. Coordination of executive functions in monolingual and bilingual children. J. Exp. Child Psychol. 2011, 110, 461-468. [CrossRef] [PubMed]
  44. Magnotti, J.F.; Beauchamp, M.S. The noisy encoding of disparity model of the McGurk effect. Psychon. Bull. Rev. 2015, 22, 701-709. [CrossRef] [PubMed]
  45. Brain Sci. 2018, 8, 85 14 of 14
  46. Basu Mallick, D.; Magnotti, J.F.; Beauchamp, M.S. Variability and stability in the McGurk effect: Contributions of participants, stimuli, time, and response type. Psychon. Bull. Rev. 2015, 22, 1299-1307. [CrossRef] [PubMed]
  47. Nath, A.R.; Beauchamp, M.S. A neural basis for interindividual differences in the McGurk effect, a multisensory speech illusion. Neuroimage 2012, 59, 781-787. [CrossRef] [PubMed]
  48. Stevenson, R.A.; Zemtsov, R.K.; Wallace, M.T. Individual differences in the multisensory temporal binding window predict susceptibility to audiovisual illusions. J. Exp. Psychol. Hum. Percept. Perform. 2012, 38, 1517-1529. [CrossRef] [PubMed]
  49. Mongillo, E.A.; Irwin, J.R.; Whalen, D.H.; Klaiman, C.; Carter, A.S.; Schultz, R.T. Audiovisual processing in children with and without autism spectrum disorders. J. Autism Dev. Disord. 2008, 38, 1349-1358. [CrossRef] [PubMed]
  50. Irwin, J.R.; Tornatore, L.A.; Brancazio, L.; Whalen, D.H. Can children with autism spectrum disorders "hear" a speaking face? Child Dev. 2011, 82, 1397-1403. [CrossRef] [PubMed]
  51. Strand, J.; Cooperman, A.; Rowe, J.; Simenstad, A. Individual differences in susceptibility to the McGurk effect: Links with lipreading and detecting audiovisual incongruity. J. Speech Lang. Hear. Res. 2014, 57, 2322-2331. [CrossRef] [PubMed]
  52. Sekiyama, K.; Soshi, T.; Sakamoto, S. Enhanced audiovisual integration with aging in speech perception: A heightened McGurk effect in older adults. Front. Psychol. 2014, 5, 323. [CrossRef] [PubMed]
  53. Walker, S.; Bruce, V.; O'Malley, C. Facial identity and facial speech processing: Familiar faces and voices in the McGurk effect. Percept. Psychophys. 1995, 57, 1124-1133. [CrossRef] [PubMed]
  54. Bovo, R.; Ciorba, A.; Prosser, S.; Martini, A. The McGurk phenomenon in Italian listeners. Acta Otorhinolaryngol. Ital. 2009, 29, 203-208. [PubMed]
  55. Fuster-Duran, A. Perception of conflicting audio-visual speech: An examination across Spanish and German. In Speechreading by Humans and Machines; Springer: Berlin/Heidelberg, Germany, 1996; pp. 135-143.
  56. Sekiyama, K. Cultural and linguistic factors in audiovisual speech processing: The McGurk effect in Chinese subjects. Percept. Psychophys. 1997, 59, 73-80. [CrossRef] [PubMed]
  57. Aloufy, S.; Lapidot, M.; Myslobodsky, M. Differences in susceptibility to the "blending illusion" among native Hebrew and English speakers. Brain Lang. 1996, 53, 51-57. [CrossRef] [PubMed]
  58. Price, C.; Thierry, G.; Griffiths, T. Speech-specific auditory processing: Where is it? Trends Cogn. Sci. 2005, 9, 271-276. [CrossRef] [PubMed]
  59. Magnotti, J.F.; Beauchamp, M.S. A causal inference model explains perception of the McGurk effect and other incongruent audiovisual speech. PLoS Comput. Biol. 2017, 13, e1005229. [CrossRef] [PubMed]
  60. Navarra, J.; Alsius, A.; Velasco, I.; Soto-Faraco, S.; Spence, C. Perception of audiovisual speech synchrony for native and non-native language. Brain Res. 2010, 1323, 84-93. [CrossRef] [PubMed]
  61. Shams, L.; Kamitani, Y.; Shimojo, S. Visual illusion induced by sound. Cogn. Brain Res. 2002, 14, 147-152. [CrossRef]
  62. Grosjean, F. Bilingual: Life and Reality; Harvard University Press: Cambridge, MA, USA, 2010; ISBN 9780674048874.
  63. Maguire, E.A.; Gadian, D.G.; Johnsrude, I.S.; Good, C.D.; Ashburner, J.; Frackowiak, R.S.J.; Frith, C.D. Navigation-related structural change in the hippocampi of taxi drivers. Proc. Natl. Acad. Sci. USA 2000, 97, 4398-4403. [CrossRef] [PubMed]
  64. Green, C.S.; Bavelier, D. Action video game modifies visual selective attention. Nature 2003, 423, 534-537. [CrossRef] [PubMed]

Related papers

The Development of Auditory-Visual Speech Perception across Languages and Age
Kaoru Sekiyama

acoustics.asn.au

To understand the now well-established auditory-visual nature of speech perception, it is necessary to understand how it develops. We know that young infants perceive speech auditory-visually by the fact that they perceive the auditory-visual illusion known as the McGurk effect; that visual information use increases over age in Englishlanguage children; and that Japanese-language adults use less visual information than do their English-language counterparts. Here we complete the developmental scene ...

View PDFchevron_right
Neural correlates of audiovisual speech processing in a second language
Christophe Pallier

Brain and Language, 2013

Neuroimaging studies of audiovisual (AV) speech processing have exclusively addressed the listeners' native language. Yet, several behavioural studies now show that AV processing plays an important role during non-native speech perception. The current fMRI study examined audiovisual speech processing in the second language of bilingual (Spanish/English) speakers. Participants were presented with auditory, visual, audiovisual congruent and audiovisual incongruent utterances in their native (L1) and non-native (L2) language. A first analysis restricted to multisensory regions whose response to the AV congruent condition was higher than either unimodal condition highlighted the posterior superior temporal sulcus (pSTS) in L1 as well as in L2. Differences in AV processing as a function of language background were not expressed in this region. However, unimodal regions in the occipital lobe had a stronger response to congruent than to incongruent AV speech in the non-native language, as compared to native. This fits well with a framework in which pSTS integrates the multimodal input in close collaboration with unimodal regions. According to these results, language background differences are predominantly expressed in these unimodal regions, whereas multisensory area pSTS is equally involved in AV integration regardless of language proficiency (L1 or L2).

View PDFchevron_right
Brain activity during audiovisual speech perception: an fMRI study of the McGurk effect
Jeffery Jones

Neuroreport, 2003

fMRI was used to assess the relationship between brain activation and the degree of audiovisual integration of speech information during a phoneme categorization task. Twelve subjects heard a speaker say the syllable /aba/ paired either with video of the speaker saying the same consonant or a di¡erent one (/ava/). In order to manipulate the degree of audiovisual integration, the audio was either synchronous or 7 400 ms out of phase with the visual stimulus. Subjects reported whether they heard the consonant /b/ or another consonant; fewer /b/ responses when the audio and visual stimuli were mismatched indicated higher levels of visual in£uence on speech perception (McGurk e¡ect). Active brain regions during presentation of the incongruent stimuli included the superior temporal and inferior frontal gyrus, as well as extrastriate, premotor and posterior parietal cortex. A regression analysis related the strength of the McGurk e¡ect to levels of brain activation. Paradoxically, higher numbers of /b/ responses were positively correlated with activation in the left occipito-temporal junction, an area often associated with processing visual motion. This activation suggests that auditory information modulates visual processing to a¡ect perception.

View PDFchevron_right
Two cortical mechanisms support the integration of visual and auditory speech: A hypothesis and preliminary data
Kayoko Okada

Neuroscience Letters, 2009

Visual speech (lip-reading) influences the perception of heard speech. The literature suggests at least two possible mechanisms for this influence: "direct" sensory-sensory interaction, whereby sensory signals from auditory and visual modalities are integrated directly, likely in the superior temporal sulcus, and "indirect" sensory-motor interaction, whereby visual speech is first mapped onto motorspeech representations in the frontal lobe, which in turn influences sensory perception via sensorymotor integration networks. We hypothesize that both mechanisms exist, and further that lip-reading functional activations of Broca's region and the posterior planum temporale reflect the sensory-motor mechanism. We tested one prediction of this hypothesis using fMRI. We assessed whether viewing visual speech (contrasted with facial gestures) activates the same network as a speech sensory-motor integration task (listen to and then silently rehearse speech). Both tasks activated locations within Broca's area, dorsal premotor cortex, and the posterior planum temporal (Spt), and focal regions of the STS, all of which have previously been implicated in sensory-motor integration for speech. This finding is consistent with the view that visual speech influences heard speech via sensory-motor networks. Lip-reading also activated a much wider network in the superior temporal lobe than the sensory-motor task, possibly reflecting a more direct cross-sensory integration network. In face-to-face conversations, we are sensitive not only to acoustic cues in the speech signal but also to the visual cues present in a speaker's face. Lip-reading, or speech-reading, is the ability of humans to understand speech by observing the lip and mouth gestures of the speaker. Visual cues provide linguistic information beyond analysis of facial expressions and can facilitate speech perception when an auditory signal occurs in a noisy environment or is degraded by noise . Further, when audio and visual speech information are mismatched (hearing "ba" while watching someone articulate "ka"), this can induce a perceptual illusion, the McGurk effect , whereby the acoustic speech information is incorrectly perceived. It is quite clear, therefore, that auditory and visual speech information interact in perception. Recent neuroimaging work investigating the neural correlates of visual speech perception implicate auditory and language-related regions in the superior temporal lobe including the superior temporal sulcus (STS), and in some reports, primary auditory cortex (Calvert,

View PDFchevron_right
Bimodal speech: early suppressive visual effects in human auditory cortex
Alexandra Fort

European Journal of Neuroscience, 2004

While everyone has experienced that vision of lip movements may improve speech perception, little is known about the neural mechanisms by which audiovisual speech information is combined. Event-related potentials (ERPs) were recorded while subjects performed an auditory recognition task among four different natural syllables randomly presented in the auditory (A), visual (V) or congruent bimodal (AV) condition. We found that (1) bimodal syllables were identified more rapidly than auditory-alone stimuli; (2) this behavioural facilitation was associated with crossmodal [AV-(A+V)] ERP effects around 120-190 ms latency, mainly expressed as a decrease of unimodal N1 generator activities in the auditory cortex. This finding provides evidence for suppressive, speech-specific audiovisual integration mechanisms, which are likely to be related to the dominance of the auditory modality for speech perception. Furthermore, the latency of the effect indicates that integration operates at pre-representational stages of stimulus analysis, probably via feedback projections from visual and/or polymodal areas.

View PDFchevron_right

Related topics

  • Psychology
  • Medicine

    • Cited by

    Behavioral Pattern Analysis between Bilingual and Monolingual Listeners’ Natural Speech Perception on Foreign-Accented English Language Using Different Machine Learning Approaches
    Md Manjurul Ahsan

    2021

    Speech perception in an adverse background/noisy environment is a complex and challenging human process, which is made even more complicated in foreign-accented language for bilingual and monolingual individuals. Listeners who have difficulties in hearing are affected most by such a situation. Despite considerable efforts, the increase in speech intelligibility in noise remains elusive. Considering this opportunity, this study investigates Bengali–English bilinguals and native American English monolinguals’ behavioral patterns on foreign-accented English language considering bubble noise, gaussian or white noise, and quiet sound level. Twelve regular hearing participants (Six Bengali–English bilinguals and Six Native American English monolinguals) joined in this study. Statistical computation shows that speech with different noise has a significant effect (p = 0.009) on listening for both bilingual and monolingual under different sound levels (e.g., 55 dB, 65 dB, and 75 dB). Here, s...

    View PDFchevron_right
    Predictors of language proficiency and cultural identification in heritage bilinguals
    Ashley Chung-Fat-Yim, Viorica Marian

    Frontiers in Communication, 2022

    According to the 2020 U.S. Census Bureau, more than 66 million residents over the age of 5 in the United States speak a language other than English at home. Some bilinguals become dominant in the majority language that is spoken in the community as opposed to their native “heritage” language acquired at home. The objective of the current study was to uncover the predictors of language proficiency and cultural identification in different groups of heritage speakers. In our sample, heritage speakers acquired their heritage language first and English second and rated their proficiency in their heritage language lower than in English. We found that English proficiency was most reliably predicted by the duration of heritage language immersion, while heritage language proficiency was most reliably predicted by contexts of acquisition and exposure to both languages. Higher heritage language proficiency was associated with greater heritage language experience through friends and reading, less English experience through family, and later age of English acquisition. The trade-off between heritage language and English language experience was more pronounced for non-Spanish than Spanish heritage speakers. Finally, despite higher proficiency in English, cultural identification was higher with the heritage language, and was predicted by heritage language receptive proficiency and heritage language experience through family and reading. We conclude that self-reported proficiency and cultural identification differ depending on heritage speakers' native languages, as well as how the heritage language and majority language are acquired and used. Our findings highlight the importance of taking individual language history into consideration when combining different groups of heritage speakers.

    View PDFchevron_right
    Investigation of Cross-Language and Stimulus-Dependent Effects on the McGurk Effect with Finnish and Japanese Speakers and Listeners
    Kaisa Tiippana

    Brain Sciences

    In the McGurk effect, perception of a spoken consonant is altered when an auditory (A) syllable is presented with an incongruent visual (V) syllable (e.g., A/pa/V/ka/ is often heard as /ka/ or /ta/). The McGurk effect provides a measure for visual influence on speech perception, becoming stronger the lower the proportion of auditory correct responses. Cross-language effects are studied to understand processing differences between one’s own and foreign languages. Regarding the McGurk effect, it has sometimes been found to be stronger with foreign speakers. However, other studies have shown the opposite, or no difference between languages. Most studies have compared English with other languages. We investigated cross-language effects with native Finnish and Japanese speakers and listeners. Both groups of listeners had 49 participants. The stimuli (/ka/, /pa/, /ta/) were uttered by two female and male Finnish and Japanese speakers and presented in A, V and AV modality, including a McGu...

    View PDFchevron_right
    Effects of training and using an audio-tactile sensory substitution device on speech-in-noise understanding
    Tomasz Wolak

    Scientific Reports, 2022

    Understanding speech in background noise is challenging. Wearing face-masks, imposed by the COVID19-pandemics, makes it even harder. We developed a multi-sensory setup, including a sensory substitution device (SSD) that can deliver speech simultaneously through audition and as vibrations on the fingertips. The vibrations correspond to low frequencies extracted from the speech input. We trained two groups of non-native English speakers in understanding distorted speech in noise. After a short session (30-45 min) of repeating sentences, with or without concurrent matching vibrations, we showed comparable mean group improvement of 14-16 dB in Speech Reception Threshold (SRT) in two test conditions, i.e., when the participants were asked to repeat sentences only from hearing and also when matching vibrations on fingertips were present. This is a very strong effect, if one considers that a 10 dB difference corresponds to doubling of the perceived loudness. The number of sentence repetitions needed for both types of training to complete the task was comparable. Meanwhile, the mean group SNR for the audio-tactile training (14.7 ± 8.7) was significantly lower (harder) than for the auditory training (23.9 ± 11.8), which indicates a potential facilitating effect of the added vibrations. In addition, both before and after training most of the participants (70-80%) showed better performance (by mean 4-6 dB) in speech-in-noise understanding when the audio sentences were accompanied with matching vibrations. This is the same magnitude of multisensory benefit that we reported, with no training at all, in our previous study using the same experimental procedures. After training, performance in this test condition was also best in both groups (SRT ~ 2 dB). The least significant effect of both training types was found in the third test condition, i.e. when participants were repeating sentences accompanied with non-matching tactile vibrations and the performance in this condition was also poorest after training. The results indicate that both types of training may remove some level of difficulty in sound perception, which might enable a more proper use of speech inputs delivered via vibrotactile stimulation. We discuss the implications of these novel findings with respect to basic science. In particular, we show that even in adulthood, i.e. long after the classical "critical periods" of development have passed, a new pairing between a certain computation (here, speech processing) and an atypical sensory modality (here, touch) can be established and trained, and that this process can be rapid and intuitive. We further present possible applications of our training program and the SSD for auditory rehabilitation in patients with hearing (and sight) deficits, as well as healthy individuals in suboptimal acoustic situations. Access to speech reading and efficient integration of visual information with auditory input is essential for enhanced understanding of speech to people with hearing loss, who naturally receive degraded auditory cues from the environment. Meanwhile, the COVID19 pandemic imposed on us the obligation of social distancing and wearing masks covering the mouth. Both these restrictions reduce transmission of sounds and prevent access to visual cues from speech/lip-reading, thereby further aggravating these difficulties 1,2. Healthy individuals also use lipreading in daily face-to-face communication, and especially benefit from it when the acoustic context is ambiguous, such as when exposed to rapid speech, speech in a non-native language, background noise and/or

    View PDFchevron_right
    Cross-Modal Interaction Between Auditory and Visual Input Impacts Memory Retrieval
    Viorica Marian

    Frontiers in Neuroscience, 2021

    How we perceive and learn about our environment is influenced by our prior experiences and existing representations of the world. Top-down cognitive processes, such as attention and expectations, can alter how we process sensory stimuli, both within a modality (e.g., effects of auditory experience on auditory perception), as well as across modalities (e.g., effects of visual feedback on sound localization). Here, we demonstrate that experience with different types of auditory input (spoken words vs. environmental sounds) modulates how humans remember concurrently-presented visual objects. Participants viewed a series of line drawings (e.g., picture of a cat) displayed in one of four quadrants while listening to a word or sound that was congruent (e.g., “cat” or <meow>), incongruent (e.g., “motorcycle” or <vroom–vroom>), or neutral (e.g., a meaningless pseudoword or a tonal beep) relative to the picture. Following the encoding phase, participants were presented with the o...

    View PDFchevron_right
    580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved