Diverse Pose Lip-Reading Framework

IRJET, 2022

When regular sound is unavailable, lip reading is a technique for comprehending speech by visually understanding the motions of the lips, face, and tongue. It also relies on information supplied by the context, linguistic understanding, and any residual hearing. Lip reading is a difficult skill for humans. To anticipate spoken words, it is necessary to have knowledge of the underlying language as well as visual cues. To interpret spoken words, experts must have a particular amount of expertise and awareness of visual expressions. It is now possible to translate lip sequences into meaningful words using deep learning. With visual information, speech recognition in noisy contexts can be improved. Deep learning is a type of machine learning method that employs numerous layers to extract higher-level characteristics from raw input. In image processing, for example, lower layers may recognise boundaries, while higher layers may identify concepts meaningful to humans, such as digits, characters, or faces. This study will look at the advancement of lip identification and recognition technology that outperforms previously offered methods considerably. Approach for automatic word identification and recognition based on varied lip motions. We will demonstrate a method for detecting and recognising human lip expressions.

2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019)

Large-scale datasets have successively proven their fundamental importance in several research fields, especially for early progress in some emerging topics. In this paper, we focus on the problem of visual speech recognition, also known as lipreading, which has received increasing interest in recent years. We present a naturally-distributed large-scale benchmark for lipreading in the wild, named LRW-1000, which contains 1, 000 classes with 718, 018 samples from more than 2, 000 individual speakers. Each class corresponds to the syllables of a Mandarin word composed of one or several Chinese characters. To the best of our knowledge, it is currently the largest word-level lipreading dataset and also the only public large-scale Mandarin lipreading dataset. This dataset aims at covering a "natural" variability over different speech modes and imaging conditions to incorporate challenges encountered in practical applications. It has shown a large variation in this benchmark in several aspects, including the number of samples in each class, video resolution, lighting conditions, and speakers' attributes such as pose, age, gender, and make-up. Besides providing a detailed description of the dataset and its collection pipeline, we evaluate several typical popular lipreading methods and perform a thorough analysis of the results from several aspects. The results demonstrate the consistency and challenges of our dataset, which may open up some new promising directions for future work.

2006 IEEE Workshop on Multimedia Signal Processing, 2006

Visual information from a speaker's mouth region is known to improve automatic speech recognition robustness. However, the vast majority of audio-visual automatic speech recognition (AVASR) studies assume frontal images of the speaker's face. In contrast, this paper investigates extracting visual speech information from the speaker's profile view, and, to our knowledge, constitutes the first real attempt to attack this problem. As with any AVASR system, the overall recognition performance depends heavily on the visual front end. This is especially the case with profile-view data, as the facial features are heavily compacted compared to the frontal scenario. In this paper, we particularly describe our visual front end approach, and report experiments on a multi-subject, small-vocabulary, bimodal, multisensory database that contains synchronously captured audio with frontal and profile face video. Our experiments show that AVASR is possible from profile views with moderate performance degradation compared to frontal video data.

HAL (Le Centre pour la Communication Scientifique Directe), 2021

Face frontalization consists of synthesizing a frontallyviewed face from an arbitrarily-viewed one. The main contribution of this paper is a robust frontalization method that preserves non-rigid facial deformations, i.e. expressions, to perform lip reading. The method iteratively estimates the rigid transformation (scale, rotation, and translation) and the non-rigid deformation between 3D landmarks extracted from an arbitrarily-viewed face, and 3D vertices parameterized by a deformable shape model. An important merit of the method is its ability to deal with large Gaussian and non-Gaussian errors in the data. For that purpose, we use the generalized Student-t distribution. The associated EM algorithm assigns a weight to each observed landmark, the higher the weight the more important the landmark, thus favoring landmarks that are only affected by rigid head movements. We propose to use the zero-mean normalized cross-correlation (ZNCC) score to evaluate the ability to preserve facial expressions. We show that the method, when incorporated into a deep lipreading pipeline, considerably improves the word classification score on an in-the-wild benchmark. An extended version with supplemental materials can be found at https: //team.inria.fr/robotlearn/rff-vsr/ * This work has been partially supported by MIAI@Grenoble Alpes (ANR-19-P3IA-0003).

IRJET, 2022

When the audio is distorted, Audio-Video Recognition (AVR) has been introduced as a solution for speech recognition tasks, as well as a visual recognition approach for speaker verification in multi-speaker situations. The method taken by AVR systems is to use the extracted information from one modality to increase the recognition ability of the other modality by complementing the missing information. By using, a relatively simple network architecture and a considerably smaller data set for training, our proposed method improves the performance of the existing similar methods for audiovisual matching, which use 3D CNNs for feature representation. We also exemplify that an effective pair selection method can significantly increase performance.

2020

Previous studies on human-machine interaction have determined that visual information can augment the speech recognition accuracy especially in noisy surroundings. Here we show a model for predicting words from video data without audio. Although already existing models have succeeded in incorporating visual data into speech recognition, all of them contained some or the other deficiency. To overcome this, we have pre-processed the data by using the haar-cascade model [2] to detect and crop around the subject's face in all frames of the video data and then use the sequence of frames as input to the model. Our proposed lipreading classification model is unique in its usage for all ranges of speakers.

arXiv (Cornell University), 2023

Lip reading is a challenging task that has many potential applications in speech recognition, human-computer interaction, and security systems. However, existing lip reading systems often suffer from low accuracy due to the limitations of video features. In this paper, we propose a novel approach that leverages visemes, which are groups of phonetically similar lip shapes, to extract more discriminative and robust video features for lip reading. We evaluate our approach on various tasks, including word-level and sentence-level lip reading, and audiovisual speech recognition using the Arman-AV dataset, a largescale Persian corpus. Our experimental results show that our viseme based approach consistently outperforms the state-of-theart methods in all these tasks. The proposed method reduces the lipreading word error rate (WER) by 9.1% relative to the best previous method.

IEEE Transactions on Neural Networks and Learning Systems

The aim of this work is to investigate the impact of 1 crossmodal self-supervised pre-training for speech reconstruc-2 tion (video-to-audio) by leveraging the natural co-occurrence 3 of audio and visual streams in videos. We propose LipSound2 4 that consists of an encoder-decoder architecture and location-5 aware attention mechanism to map face image sequences to 6 mel-scale spectrograms directly without requiring any human 7 annotations. The proposed LipSound2 model is first pre-trained 8 on ∼2400-h multilingual (e.g., English and German) audiovisual 9 data (VoxCeleb2). To verify the generalizability of the proposed 10 method, we then fine-tune the pre-trained model on domain-11 specific datasets (GRID and TCD-TIMIT) for English speech 12 reconstruction and achieve a significant improvement on speech 13 quality and intelligibility compared to previous approaches in 14 speaker-dependent and speaker-independent settings. In addition 15 to English, we conduct Chinese speech reconstruction on the 16 Chinese Mandarin Lip Reading (CMLR) dataset to verify the 17 impact on transferability. Finally, we train the cascaded lip 18 reading (video-to-text) system by fine-tuning the generated audios 19 on a pre-trained speech recognition system and achieve the state-20 of-the-art performance on both English and Chinese benchmark 21 datasets. 22 Index Terms-Lip reading, self-supervised pre-training, speech 23 recognition, speech reconstruction. 24 I. INTRODUCTION 25 I NSPIRED by human bimodal perception [1] in which both 26 sight and sound are used to improve the comprehension of 27 speech, a lot of effort has been spent on speech processing 28 tasks by leveraging visual information, for example, integrat-29 ing simultaneous lip movement sequences into speech recogni-30 tion [2], [3], guiding neural networks in isolating target speech 31 signals with a static face image for speech separation [4], [5], 32 and grounding speech recognition with visual objects and 33 scene information [6], [7]. Multimodal audiovisual methods 34 achieve significant improvement over single modality models 35 since the visual signals are invariant to acoustic noise and

2018 IEEE International Conference on Image Processing, Applications and Systems (IPAS), 2018

Lipreading is a difficult gesture classification task. One problem in computer lipreading is speaker-independence. Speaker-independence means to achieve the same accuracy on test speakers not included in the training set as speakers within the training set. Current literature is limited on speakerindependent lipreading, the few independent test speaker accuracy scores are usually aggregated within dependent test speaker accuracies for an averaged performance. This leads to unclear independent results. Here we undertake a systematic survey of experiments with the TCD-TIMIT dataset using both conventional approaches and deep learning methods to provide a series of wholly speaker-independent benchmarks and show that the best speaker-independent machine scores 69.58% accuracy with CNN features and an SVM classifier. This is less than stateof-the-art speaker-dependent lipreading machines, but greater than previously reported in independence experiments.

Proceedings of International Conference on Auditory-Visual Speech Processing, 2010

Automatic speech recognition systems that utilise the visual modality of speech often are investigated within a speakerdependent or a multi-speaker paradigm. That is, during training the recogniser will have had prior exposure to example speech from each of the possible test speakers. In a previous paper we highlighted the danger of not using different speakers in the training and test sets, and demonstrated that, within a speakerindependent configuration, lip-reading performance degrades dramatically due to ...