Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Holistic Methods
Lip-Shape Methods
Principal Component Analysis
Independent Component Analysis
Canonical Correlation Analysis
Coinertia Analysis
Results
Conclusion
References
All Topics
Computer Science
Algorithms and Data Structures

Audiovisual speech synchrony measure: application to biometrics

Profile image of Gérard CholletGérard Chollet

2007, EURASIP Journal on Applied Signal …

https://doi.org/10.1155/2007/70186
visibility

…

description

12 pages

link

1 file

Abstract

Speech is a means of communication which is intrinsically bimodal: the audio signal originates from the dynamics of the articulators. This paper reviews recent works in the field of audiovisual speech, and more specifically techniques developed to measure the level of correspondence between audio and visual speech. It overviews the most common audio and visual speech front-end processing, transformations performed on audio, visual, or joint audiovisual feature spaces, and the actual measure of correspondence between audio and visual speech. Finally, the use of synchrony measure for biometric identity verification based on talking faces is experimented on the BANCA database.

Related papers

FaceSync: A linear operator for measuring synchronization of video facial images and audio tracks
Michele Covell

2001

FaceSync is an optimal linear algorithm that finds the degree of synchronization between the audio and image recordings of a human speaker. Using canonical correlation, it finds the best direction to combine all the audio and image data, projecting them onto a single axis. ...

View PDFchevron_right
Lip synchronization of speech
David McAllister

1997

Lip synchronization is the determination of the motion of the mouth and tongue during speech. It can be deduced from the speech signal without phonemic analysis, and irrespective of the content of the speech. Our method is based on the observation that the position of the mouth over a short interval of time can be correlated with the basic shape of the spectrum of the speech over that same interval. The spectrum is obtained from a Fast Fourier Transform (FFT) and treated like a discrete probability density function. Statistical measures called moments are used to describe the shape. For several canonical utterances, video measurements of a speaker's mouth are combined with the corresponding moments to produce continuous predictor surfaces for each of three mouth parameters: jaw position, horizontal opening between the lips and vertical opening between the lips. The method involves smoothing so it is independent of the local behavior of the spectrum.

View PDFchevron_right
Temporal Synchronization and Normalization of Speech Videos for Face Recognition
usman saeed

State of the art in Biometrics, 2011

View PDFchevron_right
Recent advances in the automatic recognition of audiovisual speech
Gerasimos Potamianos

2003

Visual speech information from the speaker's mouth region has been successfully shown to improve noise robustness of automatic speech recognizers, thus promising to extend their usability into the human computer interface. In this paper, we review the main components of audio-visual automatic speech recognition and present novel contributions in two main areas: First, the visual front end design, based on a cascade of linear image transforms of an appropriate video region-of-interest, and subsequently, audio-visual speech integration. On the later topic, we discuss new work on feature and decision fusion combination, the modeling of audio-visual speech asynchrony, and incorporating modality reliability estimates to the bimodal recognition process. We also briefly touch upon the issue of audiovisual speaker adaptation. We apply our algorithms to three multi-subject bimodal databases, ranging from small-to largevocabulary recognition tasks, recorded at both visually controlled and challenging environments. Our experiments demonstrate that the visual modality improves automatic speech recognition over all conditions and data considered, however less so for visually challenging environments and large vocabulary tasks.

View PDFchevron_right
Temporal Asymmetry in Relations of Acoustic and Visual Features of Speech
Attila Tihanyi

2007

The fine temporal structure of relations of acoustic and visual features has been investigated to improve our speech to facial animation conversion system. Mutual information of acoustic and visual features has been calculated with different time shifts. Our result shows that the movement of feature points on the face of professional lip-speakers can precede the changes of acoustic parameters even by 100 milliseconds. Considering the measured time-shifts in synchrony in our system design the quality of our speech driven animations can be improved.

View PDFchevron_right
A stereo vision lip tracking algorithm and subsequent statistical analyses of the audio-video correlation in Australian English
Roland Goecke

2004

View PDFchevron_right
Evidence of correlation between acoustic and visual features of speech
Frederic Berthommier

This paper examines the degree of correlation between lip and jaw configuration and speech acoustics. The lip and jaw positions are characterised by a system of measurements taken from video images of the speaker's face and profile, and the acoustics are represented using line spectral pair parameters and a measure of RMS energy. A correlation is found between the measured acoustic parameters and a linear estimate of the acoustics recovered from the visual data. This correlation exists despite the simplicity of the visual representation and is in rough agreement with correlations measured in earlier work by Yehia et al. using different techniques. However, analysis of the estimation errors suggests that the visual information, as parameterised in our experiment, offers only a weak constraint on the acoustics. Results are discussed from the perspective of models of early audiovisual integration.

View PDFchevron_right
Inter-speaker Synchronization in Audiovisual Database for Lip-Readable Speech to Animation Conversion
Attila Tihanyi

Lecture Notes in Computer Science, 2007

The present study proposes an inter-speaker audiovisual synchronization method to decrease the speaker dependency of our direct speech to animation conversion system. Our aim is to convert an everyday speaker's voice to lip-readable facial animation for hearing impaired users. This conversion needs mixed training data: acoustic features from normal speakers coupled with visual features from professional lip-speakers. Audio and video data of normal and professional speakers were synchronized with Dynamic Time Warping method. Quality and usefulness of the synchronization were investigated in subjective test with measuring noticeable conflicts between the audio and visual part of speech stimuli. An objective test was done also, training neural network on the synchronized audiovisual data with increasing number of speakers.

View PDFchevron_right
Audio–visual speaker identification using dynamic facial movements and utterance phonetic content
Farzad Towhidkhah, Mohammad Homayounpour

Applied Soft Computing, 2011

Please cite this article in press as: V. Asadpour, et al., Audio-visual speaker identification using dynamic facial movements and utterance phonetic content, Appl. Soft Comput.

View PDFchevron_right
Detection of auditory (cross-spectral) and auditory–visual (cross-modal) synchrony
Ken Grant

Speech Communication, 2004

Detection thresholds for temporal synchrony in auditory and auditory-visual sentence materials were obtained on normal-hearing subjects. For auditory conditions, thresholds were determined using an adaptive-tracking procedure to control the degree of temporal asynchrony of a narrow audio band of speech, both positive and negative in separate tracks, relative to three other narrow audio bands of speech. For auditory-visual conditions, thresholds were determined in a similar manner for each of four narrow audio bands of speech as well as a broadband speech condition, relative to a video image of a female speaker. Four different auditory filter conditions, as well as a broadband auditoryvisual speech condition, were evaluated in order to determine whether detection thresholds were dependent on the spectral content of the acoustic speech signal. Consistent with previous studies of auditory-visual speech recognition which showed a broad, asymmetrical range of temporal synchrony for which intelligibility was basically unaffected (audio delays roughly between -40 ms and +240 ms), auditory-visual synchrony detection thresholds also showed a broad, asymmetrical pattern of similar magnitude (audio delays roughly between -45 ms and +200 ms). No differences in synchrony thresholds were observed for the different filtered bands of speech, or for broadband speech. In contrast, detection thresholds for audio-alone conditions were much smaller (between -17 ms and +23 ms) and symmetrical. These results suggest a fairly tight coupling between a subject's ability to detect cross-spectral (auditory) and cross-modal (auditory-visual) asynchrony and the intelligibility of auditory and auditoryvisual speech materials.

View PDFchevron_right

References (41)

  1. G. Potamianos, C. Neti, J. Luettin, and I. Matthews, "Audio- visual automatic speech recognition: an overview," in Issues in Visual and Audio-Visual Speech Processing, G. Bailly, E. Vatikiotis-Bateson, and P. Perrier, Eds., chapter 10, MIT Press, Cambridge, Mass, USA, 2004.
  2. T. Chen, "Audiovisual speech processing," IEEE Signal Process- ing Magazine, vol. 18, no. 1, pp. 9-21, 2001.
  3. C. C. Chibelushi, F. Deravi, and J. S. Mason, "A review of speech-based bimodal recognition," IEEE Transactions on Multimedia, vol. 4, no. 1, pp. 23-37, 2002.
  4. J. P. Barker and F. Berthommier, "Evidence of correlation between acoustic and visual features of speech," in Proceed- ings of the 14th International Congress of Phonetic Sciences (ICPhS '99), pp. 199-202, San Francisco, Calif, USA, August 1999.
  5. H. Yehia, P. Rubin, and E. Vatikiotis-Bateson, "Quantitative as- sociation of vocal-tract and facial behavior," Speech Communi- cation, vol. 26, no. 1-2, pp. 23-43, 1998.
  6. E. Bailly-Baillière, S. Bengio, F. Bimbot, et al., "The BANCA database and evaluation protocol," in Proceedings of the 4th International Conference on Audioand Video-Based Biometric Person Authentication (AVBPA '03), vol. 2688 of Lecture Notes in Computer Science, pp. 625-638, Springer, Guildford, UK, January 2003.
  7. J. Hershey and J. Movellan, "Audio-vision: using audio-visual synchrony to locate sounds," in Advances in Neural Informa- tion Processing Systems 11, M. S. Kearns, S. A. Solla, and D. A. Cohn, Eds., pp. 813-819, MIT Press, Cambridge, Mass, USA, 1999.
  8. H. Bredin, A. Miguel, I. H. Witten, and G. Chollet, "Detect- ing replay attacks in audiovisual identity verification," in Pro- ceedings of the 31st IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '06), vol. 1, pp. 621-624, Toulous, France, May 2006.
  9. J. W. Fisher III and T. Darrell, "Speaker association with signal- level audiovisual fusion," IEEE Transactions on Multimedia, vol. 6, no. 3, pp. 406-413, 2004.
  10. G. Chetty and M. Wagner, ""Liveness" verification in audio- video authentication," in Proceedings of the 10th Australian In- ternational Conference on Speech Science and Technology (SST '04), pp. 358-363, Sydney, Australia, December 2004.
  11. R. Cutler and L. Davis, "Look who's talking: speaker detection using video and audio correlation," in Proceedings of IEEE In- ternational Conference on Multimedia and Expo (ICME '00), vol. 3, pp. 1589-1592, New York, NY, USA, July-August 2000.
  12. G. Iyengar, H. J. Nock, and C. Neti, "Audio-visual synchrony for detection of monologues in video archives," in Proceed- ings of IEEE International Conference on Multimedia and Expo (ICME '03), vol. 1, pp. 329-332, Baltimore, Md, USA, July 2003.
  13. H. J. Nock, G. Iyengar, and C. Neti, "Assessing face and speech consistency for monologue detection in video," in Proceed- ings of the 10th ACM international Conference on Multimedia (MULTIMEDIA '02), pp. 303-306, Juan-les-Pins, France, De- cember 2002.
  14. M. Slaney and M. Covell, "FaceSync: a linear operator for measuring synchronization of video facial images and audio tracks," in Advances in Neural Information Processing Systems 13, pp. 814-820, MIT Press, Cambridge, Mass, USA, 2000.
  15. D. A. Reynolds, T. F. Quatieri, and R. B. Dunn, "Speaker verifi- cation using adapted Gaussian mixture models," Digital Signal Processing, vol. 10, no. 1-3, pp. 19-41, 2000.
  16. N. Sugamura and F. Itakura, "Speech analysis and synthesis methods developed at ECL in NTT-from LPC to LSP," Speech Communications, vol. 5, no. 2, pp. 199-215, 1986.
  17. C. Bregler and Y. Konig, ""Eigenlips" for robust speech recog- nition," in Proceedings of the 19th IEEE International Confer- ence on Acoustics, Speech, and Signal Processing (ICASSP '94), vol. 2, pp. 669-672, Adelaide, Australia, April 1994.
  18. M. Turk and A. Pentland, "Eigenfaces for recognition," Journal of Cognitive Neuroscience, vol. 3, no. 1, pp. 71-86, 1991.
  19. R. Goecke and B. Millar, "Statistical analysis of the relationship between audio and video speech parameters for Australian En- glish," in Proceedings of the ISCA Tutorial and Research Work- shop on Audio Visual Speech Processing (AVSP '03), pp. 133- 138, Saint-Jorioz, France, September 2003.
  20. N. Eveno and L. Besacier, "A speaker independent "liveness" test for audio-visual biometrics," in Proceedings of the 9th Eu- ropean Conference on Speech Communication and Technology (EuroSpeech '05), pp. 3081-3084, Lisbon, Portugal, September 2005.
  21. N. Eveno and L. Besacier, "Co-inertia analysis for "liveness" test in audio-visual biometrics," in Proceedings of the 4th Inter- national Symposium on Image and Signal Processing and Anal- ysis (ISPA '05), pp. 257-261, Zagreb, Croatia, September 2005.
  22. N. Fox and R. B. Reilly, "Audio-visual speaker identifica- tion based on the use of dynamic audio and visual features," in Proceedings of the 4th International Conference on Audio- and Video-Based Biometric Person Authentication (AVBPA '03), vol. 2688 of Lecture Notes in Computer Science, pp. 743-751, Springer, Guildford, UK, June 2003.
  23. C. C. Chibelushi, J. S. Mason, and F. Deravi, "Integrated per- son identification using voice and facial features," in IEE Collo- quium on Image Processing for Security Applications, vol. 4, pp. 1-5, London, UK, March 1997.
  24. A. Hyvärinen, "Survey on independent component analysis," Neural Computing Surveys, vol. 2, pp. 94-128, 1999.
  25. D. Sodoyer, L. Girin, C. Jutten, and J.-L. Schwartz, "Speech ex- traction based on ICA and audio-visual coherence," in Pro- ceedings of the 7th International Symposium on Signal Process- ing and Its Applications (ISSPA '03), vol. 2, pp. 65-68, Paris, France, July 2003.
  26. P. Smaragdis and M. Casey, "Audio/visual independent com- ponents," in Proceedings of the 4th International Symposium on Independent Component Analysis and Blind Signal Separation (ICA '03), pp. 709-714, Nara, Japan, April 2003.
  27. ICA, http://www.cis.hut.fi/projects/ica/fastica/.
  28. Canonical Correlation Analysis. http://people.imt.liu.se/ ∼magnus/cca/.
  29. S. Dolédec and D. Chessel, "Co-inertia analysis: an alterna- tive method for studying species-environment relationships," Freshwater Biology, vol. 31, pp. 277-294, 1994.
  30. J. W. Fisher, T. Darrell, W. T. Freeman, and P. Viola, "Learn- ing joint statistical models for audio-visual fusion and segre- gation," in Advances in Neural Information Processing Systems 13, T. K. Leen, T. G. Dietterich, and V. Tresp, Eds., pp. 772-778, MIT Press, Cambridge, Mass, USA, 2001.
  31. D. Sodoyer, J.-L. Schwartz, L. Girin, J. Klinkisch, and C. Jut- ten, "Separation of audio-visual speech sources: a new ap- proach exploiting the audio-visual coherence of speech stim- uli," EURASIP Journal on Applied Signal Processing, vol. 2002, no. 11, pp. 1165-1173, 2002.
  32. L. R. Rabiner, "A tutorial on hidden Markov models and se- lected applications in speech recognition," Proceedings of the IEEE, vol. 77, no. 2, pp. 257-286, 1989.
  33. S. Bengio, "An asynchronous hidden Markov model for audio- visual speech recognition," in Advances in Neural Information Processing Systems 15, S. Becker, S. Thrun, and K. Obermayer, Eds., pp. 1213-1220, MIT Press, Cambridge, Mass, USA, 2003.
  34. H. Bredin, G. Aversano, C. Mokbel, and G. Chollet, "The biosecure talking-face reference system," in Proceed- ings of the 2nd Workshop on Multimodal User Authentication (MMUA '06), Toulouse, France, May 2006.
  35. K. Messer, J. Matas, J. Kittler, J. Luettin, and G. Maitre, "XM2VTSDB: the extended M2VTS database," in Proceedings of International Conference on Audio-and Video-Based Biomet- ric Person Authentication (AVBPA '99), pp. 72-77, Washington, DC, USA, March 1999.
  36. BT-DAVID, http://eegalilee.swan.ac.uk/.
  37. S. Garcia-Salicetti, C. Beumier, G. Chollet, et al., "BIOMET: a multimodal person authentication database including face, voice, fingerprint, hand and signature modalities," in Proceed- ings of the 4th International Conference on Audio-and Video- Based Biometric Person Authentication (AVBPA '03), pp. 845- 853, Guildford, UK, June 2003.
  38. A. F. Martin, G. R. Doddington, T. Kamm, M. Ordowski, and M. Przybocki, "The DET curve in assessment of detection task performance," in Proceedings of the 5th European Conference on Speech Communication and Technology (EuroSpeech '97), vol. 4, pp. 1895-1898, Rhodes, Greece, September 1997.
  39. M. E. Sargin, E. Erzin, Y. Yemez, and A. M. Tekalp, "Multi- modal speaker identification using canonical correlation anal- ysis," in Proceedings of the 31st IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '06), vol. 1, pp. 613-616, Toulouse, France, May 2006.
  40. Text Retrieval Conference Video Track. http://trec.nist.gov/.
  41. H. Bredin and G. Chollet, "Audio-visual speech synchrony measure for talking-face identity verification," in Proceedings of the 32nd IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '07), Honolulu, Hawaii, USA, April 2007.

Related papers

Measuring audio and visual speech synchrony: methods and applications
Gérard Chollet

IET International Conference on Visual Information …, 2006

Speech is a means of communication that is intrinsically bimodal: the audio signal originates from the dynamics of the articulators. This paper reviews recent works in the field of audiovisual speech and more specifically on techniques developed to measure the level of correspondence between audio and visual speech. It overviews the most common audio and visual speech front-end processing, transformations performed on audio, visual or joint audiovisual feature spaces and the actual measure of correspondence between audio and visual speech. Finally, applications of this specific task are described.

View PDFchevron_right
Audio-visual speech synchrony measure for talking-face identity verification
Gérard Chollet

Acoustics, Speech and Signal Processing, 2007

We investigate the use of audio-visual speech synchrony measure in the framework of identity verification based on talking faces. Two synchrony measures based on Canonical Correlation Analysis and Co-Inertia Analysis respectively are introduced and their performances are evaluated on the specific task of detecting synchronized and notsynchronized audio-visual speech sequences. The notion of higheffort impostor attacks is also introduced as a dangerous threat for current biometric system based on speaker verification and face recognition. A novel biometric modality based on synchrony measures is introduced in order to improve the overall performance of identity verification, and more specifically its robustness to replay attacks.

View PDFchevron_right
Audio-visual Identity Verification: An Introductory Overview
Gérard Chollet

Progress in nonlinear …, 2007

Verification of identity is commonly achieved by looking at the face of a person and listening to his (her) speech. Automatic means of achieving this verification has been studied for several decades. Indeed, a talking face offers many features to achieve a robust verification of identity. The current deployment of videophones drives new opportunities for a secured access to remote servers (banking, certification, call centers, etc.). The synchrony of the speech signal and lip movements is a necessary condition to check that the observed talking face has not been manipulated and/or synthesized. This overview addresses face, speaker and talking face verification, as well as face and voice transformation techniques. It is demonstrated that a dedicated impostor needs limited information from a client to fool state of the art audio-visual identity verification systems.

View PDFchevron_right
Audio-visual biometrics
Miguel Vega López

2006

Abstract Biometric characteristics can be utilized in order to enable reliable and robust-to-impostor-attacks person recognition. Speaker recognition technology is commonly utilized in various systems enabling natural human computer interaction. The majority of the speaker recognition systems rely only on acoustic information, ignoring the visual modality.

View PDFchevron_right
Detecting local audio-visual synchrony in monologues utilizing vocal pitch and facial landmark trajectories
Steven Cadavid

Procedings of the British Machine Vision Conference 2009, 2009

We describe a novel approach for determining the audio-visual synchrony of a monologue video sequence utilizing vocal pitch and facial landmark trajectories as descriptors of the audio and visual modalities, respectively. The visual component is represented by the horizontal and vertical displacement of corresponding facial landmarks between subsequent frames. These facial landmarks are acquired using the statistical modeling technique, known as the Active Shape Model (ASM). The audio component is represented by the fundamental frequency, or pitch, obtained using the subharmonic-to-harmonic ratio (SHR). The synchrony between the audio and visual feature vectors is computed using Gaussian mutual information. The raw synchrony estimates obtained using this method may contain spurious synchrony values due to over-sensitivity. A filtering method is employed for discarding synchrony values that occur during non-associated audio/visual events. The human visual system is capable of distinguishing rigid and non-rigid motion of an articulator during speech. In an attempt to emulate this process, we separate rigid and non-rigid motion and compute the synchrony attributed to each. Experiments are conducted on a dataset of monologue video clip pairs. Each pair is composed of an asynchronous and synchronous version of the video clip. For the asynchronous video clips, the audio signal is displaced with respect to the visual signal. Experimental results indicate that the proposed approach is successful in detecting facial regions that demonstrate synchrony, and in distinguishing between synchronous and asynchronous sequences.

View PDFchevron_right

Related topics

  • Signal Processing on Graphs
  • Electrical and Electronic Engine...

    • Cited by

    Talking-Face Identity Verification, Audiovisual Forgery, and Robustness Issues
    Gérard Chollet

    EURASIP Journal on Advances in Signal Processing, 2009

    The robustness of a biometric identity verification (IV) system is best evaluated by monitoring its behavior under impostor attacks. Such attacks may include the transformation of one, many, or all of the biometric modalities. In this paper, we present the transformation of both speech and visual appearance of a speaker and evaluate its effects on the IV system. We propose MixTrans, a novel method for voice transformation. MixTrans is a mixture-structured bias voice transformation technique in the cepstral domain, which allows a transformed audio signal to be estimated and reconstructed in the temporal domain. We also propose a face transformation technique that allows a frontal face image of a client speaker to be animated. This technique employs principal warps to deform defined MPEG-4 facial feature points based on determined facial animation parameters (FAPs). The robustness of the IV system is evaluated under these attacks.

    View PDFchevron_right
    580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved