Recent developments in automated lip-reading

2004

2012

Abstract Computer lip-reading is one of the great signal processing challenges. Not only is the signal noisy, it is variable. However it is almost unknown to compare the performance with human lip-readers. Partly this is because of the paucity of human lip-readers and partly because most automatic systems only handle data that are trivial and therefore not representative of human speech.

International Journal of Engineering Research and, 2020

LipReading is the process of interpreting spoken word by observing lip movement, without or with audio (Noisy). The input for the application will be frames from an entire video. The application needs to detect the face (lips) from the entire frame also, trace and understand the feature patterns of the lip moments over the parameter of time. This can be done using Computer Vision (feature extraction) and Deep Convolutional Neural Network (CNN) Model Lipreading system is difficult to implement due to complex image processing, hard-to-train classifiers and long-term recognition processes. Automatic lipreading technology is a very important component of human-computer interaction technology. It is very important for human language communication and visual perception. Traditional lipreading systems usually consist of two stages: feature extraction and classification. For the first stage, a lot of methods use pixel values extracted from the mouth region of interest (ROI) represented as visual information. At present, deep learning has made significant progress in the field of computer vision (image representation, target detection, human behaviour recognition and video recognition). Therefore, automatic lipreading technology has shifted from the traditional manual feature extraction classification methods to end-to-end deep learning architecture models.

2000

We have recently proposed a new algorithm for the automatic extraction of lip feature points. Based on their positions, parameters describing the shape of the mouth are derived. Since the algorithm is based on a stereo vision face tracking system, all measurements are in real-world distances. In this paper, we evaluate the accuracy of the automatic feature extraction algorithm by comparing its results with a manual feature extraction process. The results show an average error of about 1-2mm for the internal mouth width and height. In the second part of the paper, we present the design of an AV speech database for Australian English for future experiments on the correlation of audio and video speech signals.

We investigate the performance of a machine-based lip-reading system using both shape-only parameters and full shape and ap- pearance parameters. Furthermore, we contrast the performance of a machine-based lip-reading system with human lip-reading ability. We find that the automated system outperforms human lip-readers. Curiously however, for relatively simple tasks there is little improvement in recognition accuracy when adding full ap- pearance features to the machine-based system, whereas for human lip-readers we observe significant improvements in performance. Finally, we measure the effect of ‘speaker training’ on human lip-reading ability and we find even very limited training is sufficient to improve performance.

2003

We present here a new freely available audio-visual speech database. Contrary to other existing corpora, the LIUM-AVS corpus was recorded in conditions we qualify as natural, which are, according to us, much closer to real application conditions than other databases. This database was recorded without artificial lighting using an analog camcorder in camera mode. Images were stored digitally with no compression to keep the highest possible image quality. The LIUM-AVS database comprises two parts: PBS Phonetically Balanced Sentences in French LET Spelled letters (also in French) These two parts contain sequences with both natural and blue lips. The whole database is released mainly to test and compare lip segmentation approaches on natural images, but speech recognition experiments may also be carried using this corpus. For information on obtaining the LIUM-AVS database, please contact us through our webpage (http://www-lium.univlemans.fr/lium/avs-database).

Proceedings of 13th International Conference on Pattern Recognition, 1996

This paper describes a robust method for extracting visual speech information from the shape of lips to be used for an automatic speechreading (lipreading) systems. Lip deformation is modelled by a statistically based deformable contour model which learns typical lip deformation from a training set. The main difficulty in locating and tracking lips consists of finding dominant image features for representing the lip contours. We describe the use of a statistical profile model which learns dominant image features from a training set. The model captures global intensity variation due to different illumination and different skin reflectance as well as intensity changes at the inner lip contour due to mouth opening and visibility of teeth and tongue. The method is validated for locating and tracking lip movements on a database of a broad variety of speakers.