Academia.eduAcademia.edu

Figure 8 – uploaded by Francis Rousseaux

See full PDFdownloadDownload figure
Figure 9: Confusion matrices during blind tests for respectively independent identification (on the left) and side by side discrimina- tion (on the right) of artificially generated and human tracked mouth animations. Each value have been transformed into percentages. Figure 8: Success rate during blind side by side discrimination test of artificially generated and human tracked mouth animations. Re- sults are displayed per independent discrimination criterion: gender (in blue), age (in green) and exposure to 3D animations (in red). Each box-plot describes the maximum, first quartile, median, third quartile and minimum of the candidates’ success rate.

Figure 9 Confusion matrices during blind tests for respectively independent identification (on the left) and side by side discrimina- tion (on the right) of artificially generated and human tracked mouth animations. Each value have been transformed into percentages. Figure 8: Success rate during blind side by side discrimination test of artificially generated and human tracked mouth animations. Re- sults are displayed per independent discrimination criterion: gender (in blue), age (in green) and exposure to 3D animations (in red). Each box-plot describes the maximum, first quartile, median, third quartile and minimum of the candidates’ success rate.

Related Figures (11)

Figure 1: Personal Assistant based on our text-driven mouth animation system in a workspace setup.
Figure 1: Personal Assistant based on our text-driven mouth animation system in a workspace setup.
Figure 2: The first line represents the successive 40ms Mel-spectrograms computed from a testing video audio track with the corresponding inferred 3D mouth landmarks on the second line which have been used to generate the final rendering on the last line.
Figure 2: The first line represents the successive 40ms Mel-spectrograms computed from a testing video audio track with the corresponding inferred 3D mouth landmarks on the second line which have been used to generate the final rendering on the last line.
Figure 3: Examples of extracted landmarks in 3D coordinates using FAN algorithms on cropped faces.
Figure 3: Examples of extracted landmarks in 3D coordinates using FAN algorithms on cropped faces.
Figure 4: Illustration of the 20 points composing the mouth skele- ton with their corresponding FAN annotations.
Figure 4: Illustration of the 20 points composing the mouth skele- ton with their corresponding FAN annotations.
Table 1: ADMA-Net architecture is composed of three stages. First one is in charge of learning frequency correlations at each time step whereas the second stage learns their dynamics according to the incoming Mel-spectrogram. Finally, the last stage learns a regression between these frequency dynamic feature maps and the mouth landmark positions. those is a crucial step because of cumulative approximation errors throughout the entire model. During development, we tried different approaches for the pipeline. One of them was to connect our CNN regressor directly to the Tacotron2 output. However, the output resolution was not sufficient to allow our model to learn the mouth landmarks properly. The Mel-spectrogram use by our architecture is computed using an STFT on the waveform produced by WaveGlow, as illustrated in Figure 6, to allow control on its resolution.
Table 1: ADMA-Net architecture is composed of three stages. First one is in charge of learning frequency correlations at each time step whereas the second stage learns their dynamics according to the incoming Mel-spectrogram. Finally, the last stage learns a regression between these frequency dynamic feature maps and the mouth landmark positions. those is a crucial step because of cumulative approximation errors throughout the entire model. During development, we tried different approaches for the pipeline. One of them was to connect our CNN regressor directly to the Tacotron2 output. However, the output resolution was not sufficient to allow our model to learn the mouth landmarks properly. The Mel-spectrogram use by our architecture is computed using an STFT on the waveform produced by WaveGlow, as illustrated in Figure 6, to allow control on its resolution.
Figure 6: The overall architecture is composed of successive neural modules. The modules Taoctron2 and WaveGlow are in charge of the text to speech transformation. The following ADMANet module computes the 3D mouth landmarks animation based on the speech waveform. Finally, a rendering engine generates the final animation with its corresponding audio file.
Figure 6: The overall architecture is composed of successive neural modules. The modules Taoctron2 and WaveGlow are in charge of the text to speech transformation. The following ADMANet module computes the 3D mouth landmarks animation based on the speech waveform. Finally, a rendering engine generates the final animation with its corresponding audio file.
Figure 7: Success rate during blind independent identification test of artificially generated and human tracked mouth animations. Re- sults are displayed per independent discrimination criterion: gender (in blue), age (in green) and exposure to 3D animations (in red). Each box-plot describes the maximum, first quartile, median, third quartile and minimum of the candidates’ success rate. According to Figure 7, and Figure 8, gender and age do not seem to impact candidates’ ability to solve the identification and classification tasks. Contrary, the exposure criteria tend to help the users. Therefore, as illustrated in Figure 9, both experiments confirm that our model can produce mouth animations realistic enough to fool human candidates.
Figure 7: Success rate during blind independent identification test of artificially generated and human tracked mouth animations. Re- sults are displayed per independent discrimination criterion: gender (in blue), age (in green) and exposure to 3D animations (in red). Each box-plot describes the maximum, first quartile, median, third quartile and minimum of the candidates’ success rate. According to Figure 7, and Figure 8, gender and age do not seem to impact candidates’ ability to solve the identification and classification tasks. Contrary, the exposure criteria tend to help the users. Therefore, as illustrated in Figure 9, both experiments confirm that our model can produce mouth animations realistic enough to fool human candidates.
Figure 10: Histograms of speech duration estimation for audio only (in blue) versus audio with visual feedback (in red). Each histogram corresponds to a different speech. Speech duration varies from 3 to 18s. Original answer for the speech duration (in green) and histogram tendency curve can be observed on the graph. In Figure 11, results show a small difference between candi- dates estimation errors with and without visual feedback. Compared to audio only, visual feedback slightly improves candidates ability to estimate the recordings’ duration. For more details, we provide a histogram for each question Figure 10. These histograms show that visual feedback tends to increase candidates accuracy for the du- ration estimation task. Answers are less sparse and more centered around the right one than with audio only.
Figure 10: Histograms of speech duration estimation for audio only (in blue) versus audio with visual feedback (in red). Each histogram corresponds to a different speech. Speech duration varies from 3 to 18s. Original answer for the speech duration (in green) and histogram tendency curve can be observed on the graph. In Figure 11, results show a small difference between candi- dates estimation errors with and without visual feedback. Compared to audio only, visual feedback slightly improves candidates ability to estimate the recordings’ duration. For more details, we provide a histogram for each question Figure 10. These histograms show that visual feedback tends to increase candidates accuracy for the du- ration estimation task. Answers are less sparse and more centered around the right one than with audio only.
Figure 11: Average error of the speech duration estimation by the candidates during comprehension test between audio only (in blue) and audio with visual feedback (in red). Each box-plot describes the maximum, first quartile, median, third quartile and minimum of the candidates’ duration estimation difference from the right answer.
Figure 11: Average error of the speech duration estimation by the candidates during comprehension test between audio only (in blue) and audio with visual feedback (in red). Each box-plot describes the maximum, first quartile, median, third quartile and minimum of the candidates’ duration estimation difference from the right answer.
Related topics:
Computer ScienceHuman Computer InteractionPerceptionAnimationAuditory DisplayPersonal Assistants
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved