Current trends in joint audio-video signal processing: a review

Proceedings of the IEEE, 2000

Microphones and cameras have been extensively used to observe and detect human activity and to facilitate natural modes of interaction between humans and intelligent systems. Human brain processes the audio and video modalities extracting complementary and robust information from them. Intelligent systems with audiovisual sensors should be capable of achieving similar goals. The audiovisual information fusion strategy is a key component in designing such systems. In this paper we exclusively survey the fusion techniques used in various audiovisual information fusion tasks. The fusion strategy used tends to depend mainly on the model, probabilistic or otherwise, used in the particular task to process sensory information to obtain higher level semantic information. The models themselves are task oriented. In this paper we describe the fusion strategies and the corresponding models used in audiovisual tasks such as speech recognition, tracking, biometrics, affective state recognition and meeting scene analysis. We also review the challenges and existing solutions and also unresolved or partially resolved issues in these fields. Specifically, we discuss established and upcoming work in hierarchical fusion strategies and crossmodal learning techniques, identifying these as critical areas of research in the future development of intelligent systems.

The low level acoustico-visual association reported by Yehia et al. (Speech Comm., 26(1):23-43, 1998) is exploited for audio-visual speech enhancement with natural video sequences. The aim of this study is to demonstrate that the redundant components of AV speech are extractible with a suitable representation which does not involve any categorization process. A comparative study is achieved between different types of audio features, including the initial Line Spectral Pairs (LSP) and 4-subbands envelope energy. A gain measure of the enhancement is applied for the comparison. The results clearly show that the coarse envelope features allows a better gain than the LSP.

In the audiovisual indexing context, we propose a system for automatic association of voices and images. This association can be used as a preprocessing step for existing applications like person identification systems. We use a fusion of audio and video indexes (without any prior knowledge) in order to make the information brought by each of them more robust. If both audio and video indexes are correctly segmented, this automatic association yields excellent results. In order to deal with oversegmentation, we propose an approach which uses one index to improve the segmentation of the other. We show that the use of the audio index improves an oversegmented video index on a corpus composed of French TV broadcasts.

Progress in nonlinear …, 2007

Verification of identity is commonly achieved by looking at the face of a person and listening to his (her) speech. Automatic means of achieving this verification has been studied for several decades. Indeed, a talking face offers many features to achieve a robust verification of identity. The current deployment of videophones drives new opportunities for a secured access to remote servers (banking, certification, call centers, etc.). The synchrony of the speech signal and lip movements is a necessary condition to check that the observed talking face has not been manipulated and/or synthesized. This overview addresses face, speaker and talking face verification, as well as face and voice transformation techniques. It is demonstrated that a dedicated impostor needs limited information from a client to fool state of the art audio-visual identity verification systems.

Proceedings of 2004 International Symposium on Intelligent Multimedia, Video and Speech Processing, 2004.

This paper proposes a multiple-source multiple-sample fusion approach to identity verification. Fusion is performed at two levels: intramodal and intermodal. In intramodal fusion, the scores of multiple samples (e.g. utterances or video shots) obtained from the same modality are linearly combined, where the combination weights are dependent on the difference between the score values and a user-dependent reference score obtained during enrollment. This is followed by intermodal fusion in which the means of intramodal fused scores obtained from different modalities are fused. The final fused score is then used for decision making. This twolevel fusion approach was applied to audiovisual biometric authentication, and experimental results based on the XM2VTSDB corpus show that the proposed fusion approach can achieve an error rate reduction of up to 83%.

Journal on Multimodal User Interfaces, 2018

To date, multimodal speech recognition systems based on the processing of audio and video signals show significantly better results than their unimodal counterparts. In general, researchers divide the solution of the audiovisual speech recognition problem into two parts. First, in extracting the most informative features from each modality and second, in the most successful way of fusion both modalities. Ultimately, this leads to an improvement in the accuracy of speech recognition. Almost all modern studies use this approach with video data of a standard recording speed of 25 frames per second. The choice of such a recording speed is easily explained, since the vast majority of existing audiovisual databases are recorded with this rate. However, it should be noticed that the number of 25 frames per second is a world standard for many areas and has never been specifically calculated for speech recognition tasks. The main purpose of this study is to investigate the effect brought by the high-speed video data (up to 200 frames per second) on the speech recognition accuracy. And also to find out whether the use of a high-speed video camera makes the speech recognition systems more robust to acoustical noise. To this end, we recorded a database of audiovisual Russian speech with high-speed video recordings, which consists of records of 20 speakers, each of them pronouncing 200 phrases of continuous Russian speech. Experiments performed on this database showed an improvement in the absolute speech recognition rate up to 3.10%. We also proved that the use of the high-speed camera with 200 fps allows achieving better recognition results under different acoustically noisy conditions (signal-to-noise ratio varied between 40 and 0 dB) with different types of noise (e.g. white noise, babble noise).

EURASIP Journal on Applied Signal …, 2007

Speech is a means of communication which is intrinsically bimodal: the audio signal originates from the dynamics of the articulators. This paper reviews recent works in the field of audiovisual speech, and more specifically techniques developed to measure the level of correspondence between audio and visual speech. It overviews the most common audio and visual speech front-end processing, transformations performed on audio, visual, or joint audiovisual feature spaces, and the actual measure of correspondence between audio and visual speech. Finally, the use of synchrony measure for biometric identity verification based on talking faces is experimented on the BANCA database.

Computer Vision and …, 1999

In this paper we investigate benefits of classifier combination (fusion) for a multimodal system for personal identity verification. The system uses frontal face images and speech. W e show that a sophisticated fusion strategy enables the system to outperform its facial and vocal modules when taken seperately. W e show that both trained linear weighted schem.es and fusion by Support Vector Machine classifier leads to a significant reduction of total error rates. Th,e complete system is tested on data from a publicly available audio-visual database ( X M 2 V T S , 295 subjects) according to a published protocol. 'From ACTS-M2VTS project, available at 2Available with the XM2VTS database http://www.ee.surrey.ac.uk/Research/VSSP/xm2vts 580 0-7695-0149-4/99 $10.00 0 1999 IEEE

Mobile Multimedia/Image Processing for Military and Security Applications, 2006

Person authentication can be strongly enhanced by the combination of different modalities. This is also true for the face and voice signals, which can be obtained with minimal inconvenience for the user. However, features from each modality can be combined at various different levels of processing and for face and voice signals the advantage of fusion depends strongly on the way they are combined. The aim of the work presented is to investigate the optimal strategy for combining voice and face modalities for signals of varying quality. The experimental data are taken from a newly acquired database using a PDA, which contains audio-visual recordings in different conditions. Voice features use mel-frequency cepstral coefficients, while the face signal is parameterised using wavelet coefficients in certain subbands. Results are presented for both early (feature-level) and late (score-level) fusion. At each level different fixed and variable weightings are used, both to weight between frames within each modality and to weight between modalities, where weights are based on some measure of signal reliability, such as the accuracy of automatic face detection or the audio signal to noise ratio. In addition, the contribution to authentication of information from different areas of the face is explored to determine a regional weighting for the face coefficients.

2007

Abstract Audiovisual speech processing has reached a stage of maturity where there are now numerous computational procedures needed to measure and assess multimodal signals. However, as is often the case, the results of these procedures are better known than the procedures themselves. This paper presents a MATLAB toolbox consisting of an extensive collection of tools we have developed over the past 10 years.