Academia.eduAcademia.edu

Outline

Title
Abstract
Related Work
Experiments and Results
Dataset and Training Settings
Results
Conclusion
References

Audio-Visual Speech Super-Resolution

Profile image of Sindhu HegdeSindhu Hegde

2021

Abstract

In this paper, we present an audio-visual model to perform speech super-resolution at large scale-factors (8× and 16×). Previous works attempted to solve this problem using only the audio modality as input and thus were limited to low scale-factors of 2× and 4×. In contrast, we propose to incorporate both visual and auditory signals to superresolve speech of sampling rates as low as 1kHz. In such challenging situations, the visual features assist in learning the content and improves the quality of the generated speech. Further, we demonstrate the applicability of our approach to arbitrary speech signals where the visual stream is not accessible. Our “pseudo-visual network” precisely synthesizes the visual stream solely from the low-resolution speech input. Extensive experiments and the demo video illustrate our method’s remarkable results and benefits over state-of-the-art audio-only speech super-resolution approaches. Figure 1: We present an audio-visual model for super-resolving v...

Related papers

High-quality Speech Synthesis Using Super-resolution Mel-Spectrogram
Evgeniy N Pavlovskiy

ArXiv, 2019

In speech synthesis and speech enhancement systems, melspectrograms need to be precise in acoustic representations. However, the generated spectrograms are over-smooth, that could not produce high quality synthesized speech. Inspired by image-to-image translation, we address this problem by using a learning-based post filter combining Pix2PixHD and ResUnet to reconstruct the mel-spectrograms together with super-resolution. From the resulting super-resolution spectrogram networks, we can generate enhanced spectrograms to produce high quality synthesized speech. Our proposed model achieves improved mean opinion scores (MOS) of 3.71 and 4.01 over baseline results of 3.29 and 3.84, while using vocoder Griffin-Lim and WaveNet, respectively.

View PDFchevron_right
ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement
Tal Remez

arXiv (Cornell University), 2022

Prior works on improving speech quality with visual input typically study each type of auditory distortion separately (e.g., separation, inpainting, video-to-speech) and present tailored algorithms. This paper proposes to unify these subjects and study Generalized Speech Enhancement, where the goal is not to reconstruct the exact reference clean signal, but to focus on improving certain aspects of speech. In particular, this paper concerns intelligibility, quality, and video synchronization. We cast the problem as audiovisual speech resynthesis, which is composed of two steps: pseudo audiovisual speech recognition (P-AVSR) and pseudo text-to-speech synthesis (P-TTS). P-AVSR and P-TTS are connected by discrete units derived from a self-supervised speech model. Moreover, we utilize self-supervised audiovisual speech model to initialize P-AVSR. The proposed model is coined ReVISE. ReVISE is the first high-quality model for in-the-wild video-to-speech synthesis and achieves superior performance on all LRS3 audiovisual enhancement tasks with a single model. To demonstrates its applicability in the real world, ReVISE is also evaluated on EasyCom, an audiovisual benchmark collected under challenging acoustic conditions with only 1.6 hours of training data. Similarly, ReVISE greatly suppresses noise and improves quality. Project page: https: //wnhsu.github.io/ReVISE.

View PDFchevron_right
A Survey on Image and Video Super Resolution with Deep Learning Models
Linda Sebastian

2021

Super Resolution is the process of creating high resolution images from low resolution images with minimum reduction in image quality. There are Single Image Super Resolution (SISR) and Video Super Resolution (VSR) methods. A Single Image Super Resolution goal is to change low resolution image to high resolution image. A Video Super Resolution comes from image super resolution and goal is to restore low resolution videos to high resolution videos. Convolutional Neural Networks (CNN) are the special type of deep neural networks. There are many deep learning methods for images and video super resolution. This is the survey on deep learning, CNN models for image and video super resolution. In video Super Resolution, different frames of the video are temporally connected. It means that position of different objects change from frame to frame. It is very essential to align objects. For aligning frames in video, there are two common techniques MEMC (motion estimation and compensation) met...

View PDFchevron_right
XCycles Backprojection Acoustic Super-Resolution
An Braeken

Sensors

The computer vision community has paid much attention to the development of visible image super-resolution (SR) using deep neural networks (DNNs) and has achieved impressive results. The advancement of non-visible light sensors, such as acoustic imaging sensors, has attracted much attention, as they allow people to visualize the intensity of sound waves beyond the visible spectrum. However, because of the limitations imposed on acquiring acoustic data, new methods for improving the resolution of the acoustic images are necessary. At this time, there is no acoustic imaging dataset designed for the SR problem. This work proposed a novel backprojection model architecture for the acoustic image super-resolution problem, together with Acoustic Map Imaging VUB-ULB Dataset (AMIVU). The dataset provides large simulated and real captured images at different resolutions. The proposed XCycles BackProjection model (XCBP), in contrast to the feedforward model approach, fully uses the iterative c...

View PDFchevron_right
Local-Global Fusion Network for Video Super-Resolution
Xianfang Sun

IEEE Access

View PDFchevron_right
Visual Speech Synthesis Using Photorealistic Images
Pedro Vizcaya

Citeseer

With the advances of multimedia technology, the synthesis of natural video scenes has become an important area focused on the reduction of the bit rate on specific channels. In this paper we propose two methods to obtain and use a codebook acquired from a training video which are later used to generate visual speech scenes with natural appearance. Two different classifiers have been explored, the LBG (Linde, Buzo, Gray) algorithm with a modified error criterion, and the SOFM (Self Organization Feature Map) algorithm. These vector quantizers act in a parametric domain. The parameters are obtained by applying a multiresolution transform to each frame of the video, therefore keeping frequency and space information.

View PDFchevron_right
Super-resolution Restoration of Facial Images in Video
Bir Bhanu

18th International Conference on Pattern Recognition (ICPR'06), 2006

Reconstruction-based super-resolution has been widely treated in computer vision. However, super-resolution of facial images has received very little attention. Since different parts of a face may have different motions in normal videos, this paper proposes a new method for enhancing the resolution of low-resolution facial image by handling the facial image non-uniformly. We divide low-resolution face image into different regions based on facial features and estimate motions of each of these regions using different motion models. Our experimental results show we can achieve better results than applying super-resolution on the whole face image uniformly.

View PDFchevron_right
A Deep Model for Super-resolution Enhancement from a Single Image
Kourosh Kiani

2020

This study presents a method to reconstruct a high-resolution image using a deep convolution neural network. We propose a deep model, entitled Deep Block Super Resolution (DBSR), by fusing the output features of a deep convolutional network and a shallow convolutional network. In this way, our model benefits from high frequency and low frequency features extracted from deep and shallow networks simultaneously. We use the residual layers in our model to make repetitive layers, increase the depth of the model, and make an end-to-end model. Furthermore, we employed a deep network in up-sampling step instead of bicubic interpolation method used in most of the previous works. Since the image resolution plays an important role to obtain rich information from the medical images and helps for accurate and faster diagnosis of the ailment, we use the medical images for resolution enhancement. Our model is capable of reconstructing a high-resolution image from low-resolution one in both medica...

View PDFchevron_right
iSeeBetter: Spatio-temporal video super-resolution using recurrent generative back-projection networks
Aman Chadha

Computational Visual Media

Recently, learning-based models have enhanced the performance of single-image super-resolution (SISR). However, applying SISR successively to each video frame leads to a lack of temporal coherency. Convolutional neural networks (CNNs) outperform traditional approaches in terms of image quality metrics such as peak signal to noise ratio (PSNR) and structural similarity (SSIM). On the other hand, generative adversarial networks (GANs) offer a competitive advantage by being able to mitigate the issue of a lack of finer texture details, usually seen with CNNs when super-resolving at large upscaling factors. We present iSeeBetter, a novel GAN-based spatio-temporal approach to video super-resolution (VSR) that renders temporally consistent super-resolution videos. iSeeBetter extracts spatial and temporal information from the current and neighboring frames using the concept of recurrent back-projection networks as its generator. Furthermore, to improve the “naturality” of the super-resolve...

View PDFchevron_right
Audio-Visual Speech Enhancement with Score-Based Generative Models
Timo Gerkmann

arXiv (Cornell University), 2023

This paper introduces an audiovisual speech enhancement system that leverages score-based generative models, also known as diffusion models, conditioned on visual information. In particular, we exploit audiovisual embeddings obtained from a self-supervised learning model that has been fine-tuned on lipreading. The layer-wise features of its transformer-based encoder are aggregated, time-aligned, and incorporated into the noise conditional score network. Experimental evaluations show that the proposed audiovisual speech enhancement system yields improved speech quality and reduces generative artifacts such as phonetic confusions with respect to the audio-only equivalent. The latter is supported by the word error rate of a downstream automatic speech recognition model, which decreases noticeably, especially at low input signal-to-noise ratios.

View PDFchevron_right

References (31)

  1. T. Afouras, J. S. Chung, and A. Zisserman. The conversation: Deep audio-visual speech enhancement. In INTERSPEECH, 2018.
  2. Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Deep audio-visual speech recognition. IEEE transactions on pattern anal- ysis and machine intelligence, 2018.
  3. Triantafyllos Afouras, Andrew Owens, Joon Son Chung, and Andrew Zisserman. Self- supervised learning of audio-visual objects from video. In European Conference on Computer Vision, 2020.
  4. Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, and et.al Battenberg. Deep speech 2: End-to-end speech recognition in english and mandarin. In Proceedings of the 33rd International Conference on International Conference on Machine Learning -Volume 48, ICML'16, page 173-182. JMLR.org, 2016.
  5. Sawyer Birnbaum, Volodymyr Kuleshov, Zayd Enam, Pang Wei W Koh, and Stefano Ermon. Temporal film: Capturing long-range sequence dependencies with feature- wise modulations. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  6. Y. Cheng, D. O'Shaughnessy, and P. Mermelstein. Statistical recovery of wideband speech from narrowband speech. IEEE Trans. Speech Audio Process., 2:544-548, 1994.
  7. J. S. Chung, A. Jamaludin, and A. Zisserman. You said that? In British Machine Vision Conference, 2017.
  8. J. S. Chung, A. Nagrani, and A. Zisserman. Voxceleb2: Deep speaker recognition. In INTERSPEECH, 2018.
  9. Soo-Whan Chung, Joon Son Chung, and Hong-Goo Kang. Perfect match: Improved cross-modal embeddings for audio-visual synchronisation. ICASSP 2019 -2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3965-3969, 2019.
  10. Shaojin Ding, Tianlong Chen, Xinyu Gong, Weiwei Zha, and Zhangyang Wang. Au- tospeech: Neural architecture search for speaker recognition, 2020.
  11. P. Ekstrand. Bandwidth extension of audio signals by spectral band replication. In Pro- ceedings of the 1st IEEE Benelux Workshop on Model Based Processing and Coding of Audio (MPCA '02, 2002.
  12. Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T. Freeman, and Michael Rubinstein. Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation. ACM Trans. Graph., 37, 2018. doi: 10.1145/3197517.3201357.
  13. A. Gray and J. Markel. Distance measures for speech processing. IEEE Transactions on Acoustics, Speech, and Signal Processing, 24:380-391, 1976.
  14. Sindhu B. Hegde, K.R. Prajwal, Rudrabha Mukhopadhyay, Vinay P. Namboodiri, and C.V. Jawahar. Visual speech enhancement without a real visual stream. In Proceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1926-1935, January 2021.
  15. Jesper Jensen and Cees Taal. An algorithm for predicting the intelligibility of speech masked by modulated noise maskers. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 24:1-1, 11 2016. doi: 10.1109/TASLP.2016.2585878.
  16. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  17. Keisuke Kinoshita, Tsubasa Ochiai, Marc Delcroix, and Tomohiro Nakatani. Improv- ing noise robust automatic speech recognition with single-channel time-domain en- hancement network. In IEEE International Conference on Acoustics, Speech and Sig- nal Processing (ICASSP), pages 7009-7013. IEEE, 2020.
  18. Volodymyr Kuleshov, S. Enam, and S. Ermon. Audio super resolution using neural networks. ICLR Workshops, abs/1708.00853, 2017.
  19. Erik Larsen and R. Aarts. Audio bandwidth extension: Application of psychoacoustics, signal processing and loudspeaker design. 2004.
  20. Junhyeok Lee and Seungu Han. Nu-wave: A diffusion probabilistic model for neural audio upsampling. INTERSPEECH, 2021.
  21. Jinyu Li, Li Deng, Yifan Gong, and Reinhold Haeb-Umbach. An overview of noise- robust automatic speech recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 22(4):745-777, 2014. doi: 10.1109/TASLP.2014.2304637.
  22. Kehuang Li, Z. Huang, Yong Xu, and Chin-Hui Lee. Dnn-based speech bandwidth expansion and its application to adding high-frequency missing features for automatic speech recognition of narrowband speech. In INTERSPEECH, 2015.
  23. Alastair H Moore, P Peso Parada, and Patrick A Naylor. Speech enhancement for robust automatic speech recognition: Evaluation using a baseline system and instrumental measures. Computer Speech & Language, 46:574-584, 2017.
  24. Andrew Owens and Alexei A. Efros. Audio-visual scene analysis with self-supervised multisensory features. In The European Conference on Computer Vision (ECCV), September 2018.
  25. Andrew Owens and Alexei A. Efros. Audio-visual scene analysis with self-supervised multisensory features. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
  26. Santiago Pascual, Antonio Bonafonte, and Joan Serrà. Segan: Speech enhancement generative adversarial network. pages 3642-3646, 08 2017. doi: 10.21437/Interspeech. 2017-1428.
  27. K R Prajwal, Rudrabha Mukhopadhyay, Vinay P. Namboodiri, and C.V. Jawahar. Learning individual speaking styles for accurate lip to speech synthesis. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  28. K R Prajwal, Rudrabha Mukhopadhyay, Vinay P. Namboodiri, and C.V. Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM International Conference on Multimedia, MM '20, page 484-492. Association for Computing Machinery, 2020. doi: 10.1145/3394171.3413532.
  29. Antony Rix, John Beerends, Michael Hollier, and Andries Hekstra. Perceptual evaluation of speech quality (pesq): A new method for speech quality assessment of telephone networks and codecs. volume 2, pages 749-752 vol.2, 2001. doi: 10.1109/ICASSP.2001.941023.
  30. Cees Taal, Richard Hendriks, R. Heusdens, and Jesper Jensen. A short-time objective intelligibility measure for time-frequency weighted noisy speech. pages 4214 -4217, 04 2010. doi: 10.1109/ICASSP.2010.5495701.
  31. Ruilin Xu, Rundi Wu, Yuko Ishiwaka, Carl Vondrick, and Changxi Zheng. Listening to sounds of silence for speech denoising. In Advances in Neural Information Processing Systems, volume 33, pages 9633-9648. Curran Associates, Inc., 2020.

Related papers

AERO: Audio Super Resolution in the Spectral Domain
Or Tal

Cornell University - arXiv, 2022

We present AERO, a audio super-resolution model that processes speech and music signals in the spectral domain. AERO is based on an encoder-decoder architecture with U-Net like skip connections. We optimize the model using both time and frequency domain loss functions. Specifically, we consider a set of reconstruction losses together with perceptual ones in the form of adversarial and feature discriminator loss functions. To better handle phase information the proposed method operates over the complex-valued spectrogram using two separate channels. Unlike prior work which mainly considers low and high frequency concatenation for audio super-resolution, the proposed method directly predicts the full frequency range. We demonstrate high performance across a wide range of sample rates considering both speech and music. AERO outperforms the evaluated baselines considering Log-Spectral Distance, ViSQOL, and the subjective MUSHRA test. Audio samples and code are available at https: //pages.cs.huji.ac.il/adiyoss-lab/aero/

View PDFchevron_right
Extreme-scale Talking-Face Video Upsampling with Audio-Visual Priors
Sindhu Hegde

Proceedings of the 30th ACM International Conference on Multimedia

View PDFchevron_right
Multimedia super-resolution via deep learning: A survey
Khizar Hayat

Digital Signal Processing, 2018

The recent phenomenal interest in convolutional neural networks (CNNs) must have made it inevitable for the super-resolution (SR) community to explore its potential. The response has been immense and in the last three years, since the advent of the pioneering work, there appeared too many works not to warrant a comprehensive survey. This paper surveys the SR literature in the context of deep learning. We focus on the three important aspects of multimedia-namely image, video and multi-dimensions, especially depth maps. In each case, first relevant benchmarks are introduced in the form of datasets and state of the art SR methods, excluding deep learning. Next is a detailed analysis of the individual works, each including a short description of the method and a critique of the results with special reference to the benchmarking done. This is followed by minimum overall benchmarking in the form of comparison on some common dataset, while relying on the results reported in various works.

View PDFchevron_right
Visual Speech Enhancement Without A Real Visual Stream
Sindhu Hegde

2021 IEEE Winter Conference on Applications of Computer Vision (WACV), 2021

In this work, we rethink the task of speech enhancement in unconstrained real-world environments. Current state-of-the-art methods use only the audio stream and are limited in their performance in a wide range of real-world noises. Recent works using lip movements as additional cues improve the quality of generated speech over "audio-only" methods. But, these methods cannot be used for several applications where the visual stream is unreliable or completely absent. We propose a new paradigm for speech enhancement by exploiting recent breakthroughs in speech-driven lip synthesis. Using one such model as a teacher network, we train a robust student network to produce accurate lip movements that mask away the noise, thus acting as a "visual noise filter". The intelligibility of the speech enhanced by our pseudo-lip approach is comparable (< 3% difference) to the case of using real lips. This implies that we can exploit the advantages of using lip movements even in the absence of a real video stream. We rigorously evaluate our model using quantitative metrics as well as human evaluations. Additional ablation studies and a demo video on our website containing qualitative comparisons and results clearly illustrate the effectiveness of our approach. We provide a demo video which clearly illustrates the effectiveness of our proposed approach on our website: http://cvit.iiit.ac.in/research/projects/cvit-projects/ visual-speech-enhancement-without-a-real-visual-stream. The code and models are also released for future research: https://github.com/Sindhu-Hegde/ pseudo-visual-speech-denoising.

View PDFchevron_right
Deep Networks for Image and Video Super-Resolution
Kuldeep Purohit

2022

Efficiency of gradient propagation in intermediate layers of convolutional neural networks is of key importance for super-resolution task. To this end, we propose a deep architecture for single image super-resolution (SISR), which is built using efficient convolutional units we refer to as mixed-dense connection blocks (MDCB). The design of MDCB combines the strengths of both residual and dense connection strategies, while overcoming their limitations. To enable super-resolution for multiple factors, we propose a scale-recurrent framework which reutilizes the filters learnt for lower scale factors recursively for higher factors. This leads to improved performance and promotes parametric efficiency for higher factors. We train two versions of our network to enhance complementary image qualities using different loss configurations. We further employ our network for video super-resolution task, where our network learns to aggregate information from multiple frames and maintain spatio-t...

View PDFchevron_right
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved