Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Related Work
Architectural Design
Datasets & Training
Qualitative Results
Quantitative Results
Conclusion and Future Work
Detailed Description of Datasets
Method
Ablation Study
References

One Shot Audio to Animated Video Generation

Profile image of Srishti GoelSrishti Goel

2021

Abstract

We consider the challenging problem of audio to animated video generation. We propose a novel method OneShotAu2AV to generate an animated video of arbitrary length using an audio clip and a single unseen image of a person as an input. The proposed method consists of two stages. In the first stage, OneShotAu2AV generates the talking-head video in the human domain given an audio and a person′s image. In the second stage, the talking-head video from the human domain is converted to the animated domain. The model architecture of the first stage consists of spatially adaptive normalization based multi-level generator and multiple multilevel discriminators along with multiple adversarial and non-adversarial losses. The second stage leverages attention based normalization driven GAN architecture along with temporal predictor based recycle loss and blink loss coupled with lipsync loss, for unsupervised generation of animated video. In our approach, the input audio clip is not restricted to ...

Related papers

Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion
Xin Yu

ArXiv, 2021

We propose an audio-driven talking-head method to generate photo-realistic talking-head videos from a single reference image. In this work, we tackle two key challenges: (i) producing natural head motions that match speech prosody, and (ii) maintaining the appearance of a speaker in a large head motion while stabilizing the non-face regions. We first design a head pose predictor by modeling rigid 6D head movements with a motion-aware recurrent neural network (RNN). In this way, the predicted head poses act as the low-frequency holistic movements of a talking head, thus allowing our latter network to focus on detailed facial movement generation. To depict the entire image motions arising from audio, we exploit a keypoint based dense motion field representation. Then, we develop a motion field generator to produce the dense motion fields from input audio, head poses, and a reference image. As this keypoint based representation models the motions of facial regions, head, and background...

View PDFchevron_right
Persons facial image synthesis from audio with Generative Adversarial Networks
Suneeta Budihal

ECTI Transactions on Computer and Information Technology, 2022

Article information: This paper proposes to build a framework with Generative Adversarial Network (GANs) to synthesize a person's facial image from audio input. Image and speech are the two main sources of information exchange between two entities. In some data intensive applications, a large amount of audio has to be translated into an understandable image format, with automated system, without human interference. This paper provides an end-to-end model for intelligible image reconstruction from an audio signal. The model uses a GAN architecture, which generates image features using audio waveforms for image synthesis. The model was created to produce facial images from audio of individual identities of a synthesized image of the speakers, based on the training dataset. The images of labelled persons are generated using excitation signals and the method obtained results with an accuracy of 96.88% for ungrouped data and 93.91% for grouped data.

View PDFchevron_right
TGANv2: Efficient Training of Large Models for Video Generation with Multiple Subsampling Layers
Shunta Saito

ArXiv, 2018

In this paper, we propose a novel method to efficiently train a Generative Adversarial Network (GAN) on high dimensional samples. The key idea is to introduce a differentiable subsampling layer which appropriately reduces the dimensionality of intermediate feature maps in the generator during training. In general, generators require large memory and computational costs in the latter stages of the network as the feature maps become larger, though the latter stages have relatively fewer parameters than the earlier stages. It makes training large models for video generation difficult due to the limited computational resource. We solve this problem by introducing a method that gradually reduces the dimensionality of feature maps in the generator with multiple subsampling layers. We also propose a network (Temporal GAN v2) with such layers and perform video generation experiments. As a consequence, our model trained on the UCF101 dataset at $192 \times 192$ pixels achieves an Inception S...

View PDFchevron_right
Talking Head Generation with Audio and Speech Related Facial Action Units
Jiaxing Liu

2021

The task of talking head generation is to synthesize a lip synchronized talking head video by inputting an arbitrary face image and audio clips. Most existing methods ignore the local driving information of the mouth muscles. In this paper, we propose a novel recurrent generative network that uses both audio and speech-related facial action units (AUs) as the driving information. AU information related to the mouth can guide the movement of the mouth more accurately. Since speech is highly correlated with speech-related AUs, we propose an Audio-to-AU module in our system to predict the speech-related AU information from speech. In addition, we use AU classifier to ensure that the generated images contain correct AU information. Frame discriminator is also constructed for adversarial training to improve the realism of the generated face. We verify the effectiveness of our model on the GRID dataset and TCD-TIMIT dataset. We also conduct an ablation study to verify the contribution of ...

View PDFchevron_right
Text2Video: Text-driven Talking-head Video Synthesis with Phonetic Dictionary
jiahong yuan

arXiv (Cornell University), 2021

With the advance of deep learning technology, automatic video generation from audio or text has become an emerging and promising research topic. In this paper, we present a novel approach to synthesize video from the text. The method builds a phoneme-pose dictionary and trains a generative adversarial network (GAN) to generate video from interpolated phoneme poses. Compared to audio-driven video generation algorithms, our approach has a number of advantages: 1) It only needs about 1 min of the training data, which is significantly less than audio-driven approaches; 2) It is more flexible and not subject to vulnerability due to speaker variation; 3) It significantly reduces the preprocessing and training time from several days for audio-based methods to 4 hours, which is 10 times faster. We perform extensive experiments to compare the proposed method with state-of-the-art talking face generation methods on a benchmark dataset and datasets of our own. The results demonstrate the effectiveness and superiority of our approach.

View PDFchevron_right
Talking Face Generation by Conditional Recurrent Adversarial Network
Hairong Qi

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, 2019

Given an arbitrary face image and an arbitrary speech clip, the proposed work attempts to generate the talking face video with accurate lip synchronization. Existing works either do not consider temporal dependency across video frames thus yielding abrupt facial and lip movement or are limited to the generation of talking face video for a specific person thus lacking generalization capacity. We propose a novel conditional recurrent generation network that incorporates both image and audio features in the recurrent unit for temporal dependency. To achieve both image- and video-realism, a pair of spatial-temporal discriminators are included in the network for better image/video quality. Since accurate lip synchronization is essential to the success of talking face video generation, we also construct a lip-reading discriminator to boost the accuracy of lip synchronization. We also extend the network to model the natural pose and expression of talking face on the Obama Dataset. Extensi...

View PDFchevron_right
StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2
Mohamed Elhoseiny

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Videos show continuous events, yet most-if not allvideo synthesis frameworks treat them discretely in time. In this work, we think of videos of what they should betime-continuous signals, and extend the paradigm of neural representations to build a continuous-time video generator. For this, we first design continuous motion representations through the lens of positional embeddings. Then, we explore the question of training on very sparse videos and demonstrate that a good generator can be learned by using as few as 2 frames per clip. After that, we rethink the traditional image and video discriminators pair and propose to use a single hypernetwork-based one. This decreases the training cost and provides richer learning signal to the generator, making it possible to train directly on 1024 2 videos for the first time. We build our model on top of StyleGAN2 and it is just ≈5% more expensive to train at the same resolution while achieving almost the same image quality. Moreover, our latent space features similar properties, enabling spatial manipulations that our method can propagate in time. We can generate arbitrarily long videos at arbitrary high frame rate, while prior work struggles to generate even 64 frames at a fixed rate. Our model achieves state-of-the-art results on four modern 256 2 video synthesis benchmarks and one 1024 2 resolution one. 1

View PDFchevron_right
Text2video: Text-Driven Talking-Head Video Synthesis with Personalized Phoneme - Pose Dictionary
jiahong yuan

ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

With the advance of deep learning technology, automatic video generation from audio or text has become an emerging and promising research topic. In this paper, we present a novel approach to synthesize video from the text. The method builds a phoneme-pose dictionary and trains a generative adversarial network (GAN) to generate video from interpolated phoneme poses. Compared to audio-driven video generation algorithms, our approach has a number of advantages: 1) It only needs a fraction of the training data used by an audio-driven approach; 2) It is more flexible and not subject to vulnerability due to speaker variation; 3) It significantly reduces the preprocessing, training and inference time. We perform extensive experiments to compare the proposed method with state-of-the-art talking face generation methods on a benchmark dataset and datasets of our own. The results demonstrate the effectiveness and superiority of our approach.

View PDFchevron_right
FoleyGAN: Visually Guided Generative Adversarial Network-Based Synchronous Sound Generation in Silent Videos
Sanchita Ghose

ArXiv, 2021

Deep learning based visual to sound generation systems essentially need to be developed particularly considering the synchronicity aspects of visual and audio features with time. In this research we introduce a novel task of guiding a class conditioned generative adversarial network with the temporal visual information of a video input for visual to sound generation task adapting the synchronicity traits between audiovisual modalities. Our proposed FoleyGAN model is capable of conditioning action sequences of visual events leading towards generating visually aligned realistic sound tracks. We expand our previously proposed Automatic Foley dataset to train with FoleyGAN and evaluate our synthesized sound through human survey that shows noteworthy (on average 81%) audio-visual synchronicity performance. Our approach also outperforms in statistical experiments compared with other baseline models and audio-visual datasets.

View PDFchevron_right
Diverse Generation from a Single Video Made Possible
Assaf Shocher

Cornell University - arXiv, 2021

Diverse Video Generation Spatio-Temporal Video Retargeting Conditional Video Inpainting Figure 1. We adapt classical patch-based approaches as a better, much faster non-parametric alternative to single video GANs, for a variety of video generation and manipulation tasks. As we present video results, the reader is encouraged to start from the project page. Figures only present single frame examples.

View PDFchevron_right

References (63)

  1. Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, Jie Chen, Jingdong Chen, Zhijie Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Ke Ding, Niandong Du, Erich Elsen, and Zhenyao Zhu. Deep speech 2: End-to-end speech recognition in english and mandarin. 12 2015. Deepali Aneja and Wilmot Li. Real-time lip sync for live 2d animation. 2019.
  2. Deepali Aneja, Daniel McDuff, and Shital Shah. A high-fidelity open embodied avatar with lip syncing and expression capabilities. pages 69-73, 10 2019. ISBN 978-1-4503-6860-5. doi: 10.1145/3340555.3353744.
  3. Yannis M Assael, Brendan Shillingford, Shimon Whiteson, and Nando de Freitas. Lipnet: End-to-end sentence-level lipreading. GPU Technology Conference, 2017. URL https://github.com/ Fengdalu/LipNet-PyTorch.
  4. Aayush Bansal, Shugao Ma, Deva Ramanan, and Yaser Sheikh. Recycle-gan: Unsupervised video retargeting. In ECCV, 2018.
  5. MikoÅ Ćaj BiÅ Ďkowski, DJ Sutherland, M Arbel, and A Gretton. Demystifying mmd gans. 01 2018.
  6. Matthew Brand. Voice puppetry. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co, page 21â Ȃ Ş28, 1999.
  7. Houwei Cao, David Cooper, Michael Keutmann, Ruben Gur, Ani Nenkova, and Ragini Verma. Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE transactions on affective computing, 5:377-390, 10 2014. doi: 10.1109/TAFFC.2014.2336244.
  8. Yong Cao, Wen Tien, Petros Faloutsos, and Frederic Pighin. Expressive speech-driven facial animation. ACM Trans. Graph., 24:1283-1302, 10 2005. doi: 10.1145/1095878.1095881.
  9. Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. OpenPose: realtime multi-person 2D pose estimation using Part Affinity Fields. In arXiv preprint arXiv:1812.08008, 2018. Luca Cappelletta and Naomi Harte. Phoneme-to-viseme mapping for visual speech recognition. Proceedings of the International Conference on Pattern Recognition Applications and Methods (ICPRAM 2012), 2, 05 2012.
  10. J. S. Chung and A. Zisserman. Out of time: automated lip sync in the wild. In Workshop on Multi-view Lip-reading, ACCV, 2016.
  11. Joon Son Chung, Amir Jamaludin, and Andrew Zisserman. You said that? In British Machine Vision Conference, 2017.
  12. Barker J. Cunningham S. Shao X Cooke, M. An audio-visual corpus for speech perception and auto- matic speech recognition. The Journal of the Acoustical Society of America, 120(5):2421â Ȃ Ş2424, 2006.
  13. Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil Bharath. Generative adversarial networks: An overview. IEEE Signal Processing Magazine, 35, 10 2017. doi: 10.1109/MSP.2017.2765202.
  14. Matthew Dailey, Michael Lyons, Miyuki Kamachi, H Ishi, Jiro Gyoba, and Garrison Cottrell. Cultural differences in facial expression classification. Proc. Cognitive Neuroscience Society, 9th Annual Meeting, San Francisco CA, page 153, 06 2002.
  15. Pif Edwards, Chris Landreth, Eugene Fiume, and Karan Singh. Jali: an animator-centric viseme model for expressive lip synchronization. ACM Transactions on Graphics, 35:1-11, 07 2016. doi: 10.1145/2897824.2925984.
  16. Epic Games. Unreal engine , online: https://www. unrealengine. com. 2007.
  17. MichaÃńl Gilbert, Samuel Demarchi, and Isabel Urdapilleta. Facshuman a software to create experimental material by modeling 3d facial expression. pages 333-334, 11 2018. doi: 10.1145/ 3267851.3267865.
  18. Riza Guler, Natalia Neverova, and Iasonas Kokkinos. Densepose: Dense human pose estimation in the wild. pages 7297-7306, 06 2018. doi: 10.1109/CVPR.2018.00762.
  19. David Huggins Daines, M. Kumar, A. Chan, A.W. Black, M. Ravishankar, and Alexander Rudnicky. Pocketsphinx: A free, real-time continuous speech recognition system for hand-held devices. volume 1, pages I -I, 06 2006. doi: 10.1109/ICASSP.2006.1659988.
  20. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei Efros. Image-to-image translation with conditional adversarial networks. pages 5967-5976, 07 2017. doi: 10.1109/CVPR.2017.632.
  21. Tero Karras, Timo Aila, Samuli Laine, Antti Herva, and Jaakko Lehtinen. Audio-driven facial animation by joint end-to-end learning of pose and emotion. ACM Transactions on Graphics, 36: 1-12, 07 2017. doi: 10.1145/3072959.3073658.
  22. Byung-Hak Kim and Varun Ganapath. Lumià ĺrenet: Lecture video synthesis from audio. 2016.
  23. Junho Kim, Minjae Kim, Hyeon-Woo Kang, and Kwanghee Lee. U-gat-it: Unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation, 07 2019.
  24. Ann Kring and Denise Sloan. The facial expression coding system (faces): Development, validation, and utility. Psychological assessment, 19:210-24, 06 2007. doi: 10.1037/1040-3590.19.2.210.
  25. Soonkyu Lee and Dongsuk Yook. Audio-to-visual conversion using hidden markov models. pages 563-570, 08 2002. doi: 10.1007/3-540-45683-X_60.
  26. Giuseppe Riccardo Leone, Giulio Paci, and Piero Cosi. Lucia: An open source 3d expressive avatar for multimodal h.m.i. volume 78, pages 193-202, 01 2012. doi: 10.1007/978-3-642-30214-5_21.
  27. Wesley Mattheyses and Werner Verhelst. Audiovisual speech synthesis: An overview of the state-of- the-art. Speech Communication, 66, 11 2014. doi: 10.1016/j.specom.2014.11.001.
  28. Ricard Marxer Jon Barker Najwa Alghamdi, Steve Maddock and Guy J. Brown. A corpus of audio- visual lombard speech with frontal and profile view, the journal of the acoustical society of america 143, el523 (2018); https://doi.org/10.1121/1.5042758, 2018.
  29. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An asr corpus based on public domain audio books. pages 5206-5210, 04 2015. doi: 10.1109/ICASSP.2015. 7178964.
  30. Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
  31. Isabella Poggi, Catherine Pelachaud, F. Rosis, Valeria Carofiglio, and Berardina Carolis. Greta. A Believable Embodied Conversational Agent, pages 3-25. 01 2005. doi: 10.1007/1-4020-3051-7_1.
  32. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. volume 9351, pages 234-241, 10 2015. ISBN 978-3-319-24573-7. doi: 10.1007/978-3-319-24574-4_28.
  33. Marcos Santos PÃl'rez, Eva Gonzà ąlez-Parada, and Jose Manuel Cano-Garcia. Avatar: An open source architecture for embodied conversational agents in smart environments. volume 6693, pages 109-115, 06 2011. doi: 10.1007/978-3-642-21303-8_15.
  34. A. Simons and Stephen Cox. Generation of mouthshapes for a synthetic talking head. Proceedings of the Institute of Acoustics, Autumn Meeting, 01 1990.
  35. Tereza Soukupova and Jan Cech. Real-time eye blink detection using facial landmarks, 2016.
  36. Andreea Stef, Kaveen Perera, Hubert Shum, and Edmond Ho. Synthesizing expressive facial and speech animation by text-to-ipa translation with emotion control. pages 1-8, 12 2018. doi: 10.1109/SKIMA.2018.8631536.
  37. Supasorn Suwajanakorn, Steven Seitz, and Ira Kemelmacher. Synthesizing obama: learning lip sync from audio. ACM Transactions on Graphics, 36:1-13, 07 2017. doi: 10.1145/3072959.3073640.
  38. Sarah Taylor, Moshe Mahler, Barry-John Theobald, and Iain Matthews. Dynamic units of visual speech. pages 275-284, 07 2012.
  39. Guanzhong Tian, Yi Yuan, and Yong Liu. Audio2face: Generating speech/face animation from single audio with attention-based bidirectional lstm networks. 05 2019.
  40. Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. MoCoGAN: Decomposing motion and content for video generation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1526-1535, 2018.
  41. Konstantinos Vougioukas, Stavros Petridi, and Maja Pantic. End-to-end speech-driven facial anima- tion with temporal gans. Journal of Foo, 14(1):234-778, 2004.
  42. Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. 11 2017.
  43. Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
  44. Ting-Chun Wang, Ming-Yu Liu, Andrew Tao, Guilin Liu, Jan Kautz, and Bryan Catanzaro. Few-shot video-to-video synthesis. In Conference on Neural Information Processing Systems (NeurIPS), 2019.
  45. O. Wiles, A.S. Koepke, and A. Zisserman. X2face: A network for controlling face generation by using images, audio, and pose codes. In European Conference on Computer Vision, 2018.
  46. Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky. Few-shot adversarial learning of realistic neural talking head models, 05 2019.
  47. Bolei Zhou, Aditya Khosla, Ã Ȃgata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. 12 2015.
  48. Youssef Alami Mejjati, Christian Richardt, James Tompkin, Darren Cosker, and Kwang Kim. Unsu- pervised attention-guided image to image translation, 06 2018.
  49. Yannis M Assael, Brendan Shillingford, Shimon Whiteson, and Nando de Freitas. Lipnet: End-to-end sentence-level lipreading. GPU Technology Conference, 2017. URL https://github.com/ Fengdalu/LipNet-PyTorch.
  50. Akshay Asthana, Stefanos Zafeiriou, Shiyang Cheng, and Maja Pantic. Incremental face alignment in the wild. pages 1859-1866, 06 2014. doi: 10.1109/CVPR.2014.240.
  51. MikoÅ Ćaj BiÅ Ďkowski, DJ Sutherland, M Arbel, and A Gretton. Demystifying mmd gans. 01 2018.
  52. Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. OpenPose: realtime multi-person 2D pose estimation using Part Affinity Fields. In arXiv preprint arXiv:1812.08008, 2018.
  53. Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. 12 2017.
  54. Alexandre Alahi Justin Johnson and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. 2016.
  55. Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 12 2014.
  56. Yanchun Li, Nanfeng Xiao, and Wanli Ouyang. Improved generative adversarial networks with reconstruction loss. Neurocomputing, 323, 10 2018. doi: 10.1016/j.neucom.2018.10.014.
  57. Yu Tian Mubbasir Kapadia Long Zhao, Xi Peng and Dimitris Metaxas1. Learning to forecast and refine residual motion for image-to-video generation, 2018.
  58. Wang Mei and Weihong Deng. Deep face recognition: A survey. 04 2018.
  59. Nataniel Ruiz, Eunji Chong, and James M. Rehg. Fine-grained head pose estimation without key- points. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2018.
  60. Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv 1409.1556, 09 2014.
  61. Tereza Soukupova and Jan Cech. Real-time eye blink detection using facial landmarks, 2016.
  62. Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. MoCoGAN: Decomposing motion and content for video generation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1526-1535, 2018.
  63. Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.

Related papers

Robust One Shot Audio to Video Generation
Srishti Goel

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2020

Audio to Video generation is an interesting problem that has numerous applications across industry verticals including film making, multi-media, marketing, education and others. High-quality video generation with expressive facial movements is a challenging problem that involves complex learning steps for generative adversarial networks. Further, enabling one-shot learning for an unseen single image increases the complexity of the problem while simultaneously making it more applicable to practical scenarios.In the paper, we propose a novel approach OneShotA2V to synthesize a talking person video of arbitrary length using as input: an audio signal and a single unseen image of a person. OneShotA2V leverages curriculum learning to learn movements of expressive facial components and hence generates a high-quality talking head video of the given person.Further, it feeds the features generated from the audio input directly into a generative adversarial network and it adapts to any given u...

View PDFchevron_right
Multi Modal Adaptive Normalization for Audio to Video Generation
Srishti Goel

ArXiv, 2020

Speech-driven facial video generation has been a complex problem due to its multi-modal aspects namely audio and video domain. The audio comprises lots of underlying features such as expression, pitch, loudness, prosody(speaking style) and facial video has lots of variability in terms of head movement, eye blinks, lip synchronization and movements of various facial action units along with temporal smoothness. Synthesizing highly expressive facial videos from the audio input and static image is still a challenging task for generative adversarial networks. In this paper, we propose a multi-modal adaptive normalization(MAN) based architecture to synthesize a talking person video of arbitrary length using as input: an audio signal and a single image of a person. The architecture uses the multi-modal adaptive normalization, keypoint heatmap predictor, optical flow predictor and class activation map[58] based layers to learn movements of expressive facial components and hence generates a ...

View PDFchevron_right
Realistic Video Synthesis from Audio using GAN
IJRASET Publication

International Journal for Research in Applied Science & Engineering Technology (IJRASET), 2025

Realistic video generation from audio input is a challenging and emerging domain in the intersection of natural language processing, computer vision, and generative modeling. The ability to automatically generate coherent and visually compelling video content from raw audio has promising applications in media creation, virtual education, assistive technologies, and entertainment. Manual video creation remains time-consuming and skill-intensive, while automated solutions often lack semantic alignment and visual realism. To address this gap, this project proposes an end-to-end intelligent pipeline that synthesizes realistic video content from audio input using Generative Adversarial Networks (GANs). The system begins by transcribing the user's audio using OpenAI's Whisper ASR model, followed by extracting meaningful textual descriptions via a language model (e.g., Groq LLaMA or OpenAI GPT). The script is used to generate key search terms for visual content retrieval, sourcing high-quality imagery from the Pexels API. Speech is generated using Edge TTS, and synchronized subtitles are created.The images are compiled into a dynamic video using MoviePy, and visual quality is further enhanced using Real-ESRGAN for super-resolution. The final output is a short, high-resolution, contextually accurate video with natural narration and relevant imagery. This work demonstrates the effectiveness of combining audio processing, NLP, GAN-based enhancement, and open content APIs to automate realistic video generation from scratch.

View PDFchevron_right
Automated Voice-to-Image Generation Using Generative Adversarial Networks in Machine Learning
Prasanna Yeluri

2022

Creating visuals from words may appear to be a complex process, but it is achievable with today's technological advancements in Information Systems. Naturally, all the human-centric actions and assumptions may lead to visualization using Artificial Intelligence. In today's Information Systems technological world, any item or a thing can be best described in pictorial form as a human person. Our paper aims to focus on providing machines with this intelligence. To complete this challenge, we used Natural Language Processing with Deep Learning. Our primary focus is on Generative Adversarial Networks. GANs will generate data based on word labels that are provided. NLP is also important since it helps to translate the provided speech into embedding vectors that the model can use. Our study is on the CUB dataset, which comprises bird photos. In today's world, there are text-to-image generating models accessible. The authors investigated all of them, extending text-to-image generation to voice-to-image generation.

View PDFchevron_right
NWT: Towards natural audio-to-video generation with representation learning
rayhane mama

ArXiv, 2021

In this work we introduce NWT, an expressive speech-to-video model. Unlike approaches that use domain-specific intermediate representations such as pose keypoints, NWT learns its own latent representations, with minimal assumptions about the audio and video content. To this end, we propose a novel discrete variational autoencoder with adversarial loss, dVAE-Adv, which learns a new discrete latent representation we call Memcodes. Memcodes are straightforward to implement, require no additional loss terms, are stable to train compared with other approaches, and show evidence of interpretability. To predict on the Memcode space, we use an autoregressive encoder-decoder model conditioned on audio. Additionally, our model can control latent attributes in the generated video that are not annotated in the data. We train NWT on clips from HBO’s Last Week Tonight with John Oliver. NWT consistently scores above other approaches in Mean Opinion Score (MOS) on tests of overall video naturalness...

View PDFchevron_right
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved