Realistic speech animation based on observed 3D face dynamics

Proceedings Computer Animation '98 (Cat. No.98EX169)

The importance of faces in human interaction explains the desire for synthetic faces as a communication vehicle in computer graphics. Unfortunately, animating a face is a very complex process, partly because of our very familiarity with the human face. Moreover, depending upon the application, there exists a wide range of faces to animate, whether realistically or artistically. We present here an animation system that captures live facial expressions from a performance actor, and uses them to animate in real-time a synthetic character. Our approach is based upon a bank of 3D facial expressions of a synthetic model. We also present improvements on the control of markers, on specific details for more delicate control of the eyelids, and on the use of filters applied to the sequence of markers displacements. More flexibility in the bank of expressions is provided by subdividing some of the expressions into independent components, and by controlling accessories such as teeth and the tongue only by means of the reconstructed expression. The resulting system is very flexible, intuitive to use, and the real-time animations provide immediate feedback to express and refine animator and performer skills.

2001

In this paper, we describe a new technique for expressive and realistic speech animation. We use an optical tracking system that extracts the 3D positions of markers attached at the feature point locations to capture the movements of the face of a talking person. We use the feature points as defined by the MPEG-4 standard. We then form a vector space representation by using the Principal Component Analysis of this data. We call this space "expression and viseme space". Such a representation not only offers insight into improving realism of animated faces, but also gives a new way of generating convincing speech animation and blending between several expressions. As the rigid body movements and deformation constraints on the facial movements have been considered through this analysis, the resulting facial animation is very realistic.

IEE Proceedings - Vision, Image, and Signal Processing, 2004

A system capable of producing near video-realistic animation of a speaker given only speech inputs is presented. The audio input is a continuous speech signal, requires no phonetic labelling and is speaker-independent. The system requires only a short video training corpus of a subject speaking a list of viseme-targeted words in order to achieve convincing realistic facial synthesis. The system learns the natural mouth and face dynamics of a speaker to allow new facial poses, unseen in the training video, to be synthesised. To achieve this the authors have developed a novel approach which utilises a hierarchical and nonlinear principal components analysis (PCA) model which couples speech and appearance. Animation of different facial areas, defined by the hierarchy, is performed separately and merged in post-processing using an algorithm which combines texture and shape PCA data. It is shown that the model is capable of synthesising videos of a speaker using new audio segments from both previously heard and unheard speakers.

Computers & Graphics, 2006

This paper presents a novel approach for the generation of realistic speech synchronized 3D facial animation that copes with anticipatory and perseveratory coarticulation. The methodology is based on the measurement of 3D trajectories of fiduciary points marked on the face of a real speaker during the speech production of CVCV nonsense words. The trajectories are measured from standard video sequences using stereo vision photogrammetric techniques. The first stationary point of each trajectory associated with a phonetic segment is selected as its articulatory target. By clustering according to geometric similarity all articulatory targets of a same segment in different phonetic contexts, a set of phonetic context-dependent visemes accounting for coarticulation is identified. These visemes are then used to drive a set of geometric transformation/deformation models that reproduce the rotation and translation of the temporomandibular joint on the 3D virtual face, as well as the behavior of the lips, such as protrusion, and opening width and height of the natural articulation. This approach is being used to generate 3D speech synchronized animation from both natural and synthetic speech generated by a text-to-speech synthesizer.

International Conference on Visual Information Engineering (VIE 2003). Ideas, Applications, Experience, 2003

We present a system capable of producing video-realistic videos of a speaker given audio only. The audio input signal requires no phonetic labelling and is speaker independent. The system requires only a small training set of video to achieve convincing realistic facial synthesis. The system learns the natural mouth and face dynamics of a speaker to allow new facial poses, unseen in the training video, to be synthesised. To achieve this we have developed a novel approach which utilises a hierarchical and non-linear PCA model which couples speech and appearance. We show that the model is capable of synthesising videos of a speaker using new audio segments from both previously heard and unheard speakers. The model is highly compact making it suitable for a wide range of real-time applications in multimedia and telecommunications using standard hardware.

2004

We introduce a novel approach to modeling the dynamics of human facial motion induced by the action of speech for the purpose of synthesis. We represent the trajectories of a number of salient features on the human face as the output of a dynamical system made up of two subsystems, one driven by the deterministic speech input, and a second driven by an unknown stochastic input. Inference of the model (learning) is performed automatically and involves an extension of independent component analysis to time-depentend data. Using a shape-texture decompositional representation for the face, we generate facial image sequences reconstructed from synthesized feature point positions.

2011 IEEE International Conference on Multimedia and Expo, 2011

In this paper, a system for speech-driven animation of generic 3D head models is presented. The system is based on the inversion of a joint Audio-Visual Hidden Markov Model to estimate the visual information from speech data. Estimated visual speech features are used to animate a simple face model. The animation of a more complex head model is then obtained by automatically mapping the deformation of the simple model to it. The proposed algorithm allows the animation of 3D head models of arbitrary complexity through a simple setup procedure. The resulting animation is evaluated in terms of intelligibility of visual speech through subjective tests, showing a promising performance.

Computer Animation and Virtual Worlds, 2012

We propose an integrated facial dynamics model addressing the animation of 3D humanoid faces in real time. The computational model mimics facial motion by reproducing the layered anatomical structure of a human head including the bony structure, overlapping facial muscles and the skin. The model is flexible enough to animate face meshes of various shape, connectivity and scale.

Modelling and Motion …, 1998

Performer-driven animation has been used with success [1], first of all to repro-duce human body motion. While there are different capturing hardware-software systems to map the motion of a performer on the motion of a model of the body or face, little has been done both on ...

IEEE Computer Graphics and Applications, 2000

E motional facial expressions don't relate directly to speech but reflect a person's emotional state and personality. In social interactions, they give clues to a speaker's state of mind and help people interpret the speaker's tone or meaning. To create believable virtual characters, we must address the problem of simulating natural, credible emotional facial expressions. Indeed, previous studies have shown that people perceive synthetic faces displaying no emotional expressions as unrealistic and even unpleasant.