Academia.eduAcademia.edu

Outline

Title
Abstract
Figures
Introduction
Related Works
Our Method
Results and Discussion
Evaluation of Background Texture Auralization
Conclusion and Future Work
References
All Topics
Computer Science
Information Systems

Real-time rendering of decorative sound textures for soundscapes

Profile image of Jinta ZhengJinta Zheng

2020, ACM Transactions on Graphics

https://doi.org/10.1145/3414685.3417875
visibility

…

description

12 pages

link

1 file

Abstract

Audio recordings contain rich information about sound sources and their properties such as the location, loudness, and frequency of events. One prevalent component in sound recordings is the sound texture, which contains a massive number of events. In such a texture, there can be some distinct and repeated sounds that we term as a foreground sound. Birds chirping in the wind is one such decorative sound texture with the chirping as a foreground sound and the wind as a background texture. To render these decorative sound textures in real-time and with high quality, we create two-layer Markov Models to enable smooth transitions from sound grain to sound grain and propose a hierarchical scheme to generate Head-Related Transfer Function filters for localization cues of sounds represented as area/volume sources. Moreover, during the synthesis stage, we provide control over the frequency and intensity of sounds for customization. Lastly, foreground sounds are often blended into background...

Figures (15)
Fig. 1. Our method renders a decorative sound texture of a city street during a rainstorm. The images (top row) show the virtual scene from the listener’: perspective over an eight second time period. The plots (bottom row) show the corresponding color-coded waveform of the rendered decorative sound texture in the left and right ears. Raindrops hitting the road (blue) is the background texture, raindrops hitting the umbrella (dark green) is the first foreground sound and birds chirping (light green) is the second foreground sound. All the foreground sounds and background textures were extracted from recordings at Fon et al. [2013]. The intensity of the background texture increases throughout the eight seconds, as intended by the scene designer. Additionally, the even frequency of the foreground sounds increases over time, which is also controlled by our methods. This scene is built in CARLA [Dosovitskiy et al. 2017].
Fig. 1. Our method renders a decorative sound texture of a city street during a rainstorm. The images (top row) show the virtual scene from the listener’: perspective over an eight second time period. The plots (bottom row) show the corresponding color-coded waveform of the rendered decorative sound texture in the left and right ears. Raindrops hitting the road (blue) is the background texture, raindrops hitting the umbrella (dark green) is the first foreground sound and birds chirping (light green) is the second foreground sound. All the foreground sounds and background textures were extracted from recordings at Fon et al. [2013]. The intensity of the background texture increases throughout the eight seconds, as intended by the scene designer. Additionally, the even frequency of the foreground sounds increases over time, which is also controlled by our methods. This scene is built in CARLA [Dosovitskiy et al. 2017].
a a a, Fig. 3. Background textures have a homogeneous difference in loudness to the (a) left ear and (b) right ear; while the foreground sound events (green boxes) have varying differences. In this example, the background texture is rain drops hitting the ground and the foreground sound is heavy rain drops hitting a metal roof. The recording is from Font et al. [2013].
a a a, Fig. 3. Background textures have a homogeneous difference in loudness to the (a) left ear and (b) right ear; while the foreground sound events (green boxes) have varying differences. In this example, the background texture is rain drops hitting the ground and the foreground sound is heavy rain drops hitting a metal roof. The recording is from Font et al. [2013].
(a) Decorative Sound Texture Processing (b) Decorative Sound Texture Synthesis (c) Decorative Sound Texture Auralization Fig. 4. Rendering of a decorative sound texture includes sound synthesis and auralization. (a) We provide a foreground sound extraction algorithm to create a decorative sound texture with foreground sounds and a background texture on area/volume sources in virtual scenes. (b) At run-time, for decorative sound texture, we synthesize the foreground sounds (green) and background texture (blue) with different Markov model designs. (c) For decorative sound texture auralization, we compute the convolution of the synthesized sound and HRTF filters constructed with hierarchical grids to efficiently capture the location information for the area/volume sources of foreground sounds and background textures. We mix the auralized foreground sound and background textures to generate the final sound for the listener.
(a) Decorative Sound Texture Processing (b) Decorative Sound Texture Synthesis (c) Decorative Sound Texture Auralization Fig. 4. Rendering of a decorative sound texture includes sound synthesis and auralization. (a) We provide a foreground sound extraction algorithm to create a decorative sound texture with foreground sounds and a background texture on area/volume sources in virtual scenes. (b) At run-time, for decorative sound texture, we synthesize the foreground sounds (green) and background texture (blue) with different Markov model designs. (c) For decorative sound texture auralization, we compute the convolution of the synthesized sound and HRTF filters constructed with hierarchical grids to efficiently capture the location information for the area/volume sources of foreground sounds and background textures. We mix the auralized foreground sound and background textures to generate the final sound for the listener.
Fig. 7. A synthesized ocean wave sound starting from an a priori-class state of low entropy and moving to a high entropy state. The resulting background texture smoothly follows the entropy changes controlled by the a priori-class layer of the two-layer Markov model. With the two-layer Markov models, we generate realistic decora- tive sound textures with similar features to the original recordings. We present our evaluations of the realism and quality of our deco- rative sound texture synthesis in Section 5.2.
Fig. 7. A synthesized ocean wave sound starting from an a priori-class state of low entropy and moving to a high entropy state. The resulting background texture smoothly follows the entropy changes controlled by the a priori-class layer of the two-layer Markov model. With the two-layer Markov models, we generate realistic decora- tive sound textures with similar features to the original recordings. We present our evaluations of the realism and quality of our deco- rative sound texture synthesis in Section 5.2.
Re me en ere Sami ment ane nes sneer OE IEE Fig. 8. Auralization of area/volume sources of (a) a foreground sound and (b) a background texture. (a) The foreground sound is emitted randomly from a point guided by the perception model that avoids a point (dashed arrow) where the sound is masked by a background texture. The HRTF is interpolated at the point in 2D interaural-polar coordinates using the hierarchical grid. (b) The background texture is heard from the area/volume sources and the HRTF is constructed with the ray-intersection test in the hierarchical grid.
Re me en ere Sami ment ane nes sneer OE IEE Fig. 8. Auralization of area/volume sources of (a) a foreground sound and (b) a background texture. (a) The foreground sound is emitted randomly from a point guided by the perception model that avoids a point (dashed arrow) where the sound is masked by a background texture. The HRTF is interpolated at the point in 2D interaural-polar coordinates using the hierarchical grid. (b) The background texture is heard from the area/volume sources and the HRTF is constructed with the ray-intersection test in the hierarchical grid.
AS] VA A AMT AV GIMBAL MA IEE AMP AAV GAMGLIUAS MA AES Fig. 9. Comparison of our decorative sound texture extraction to DAP [Tian et al. 2019], NMF [Spiertz and Gnann 2009], and spectral subtraction [Boll 1979] using (a) SDR, (b) SAR, and (c) SIR. We average the values of SDR, SAR, and SIR over the 5 categories (animal, natural, urban, human, and music). Our extraction results have higher average scores compared to the other methods. Our test database and results can be accessed here: https://github.com/hiebelky, JKSound-Benchmark.
AS] VA A AMT AV GIMBAL MA IEE AMP AAV GAMGLIUAS MA AES Fig. 9. Comparison of our decorative sound texture extraction to DAP [Tian et al. 2019], NMF [Spiertz and Gnann 2009], and spectral subtraction [Boll 1979] using (a) SDR, (b) SAR, and (c) SIR. We average the values of SDR, SAR, and SIR over the 5 categories (animal, natural, urban, human, and music). Our extraction results have higher average scores compared to the other methods. Our test database and results can be accessed here: https://github.com/hiebelky, JKSound-Benchmark.
Fig. 11. Synthesized background textures using McDermott et al. [2011] (second column) and our method (third column) compared to the original sound texture of (a) wind, and (b) a jackhammer. From top to bottom, the figure shows the waveform, Mel spectrogram, and cross-band envelope correlation matrix for each audio clip. Our resulting background textures reproduce closely the features of the input recordings. The sounds are from Font et al. [2013].
Fig. 11. Synthesized background textures using McDermott et al. [2011] (second column) and our method (third column) compared to the original sound texture of (a) wind, and (b) a jackhammer. From top to bottom, the figure shows the waveform, Mel spectrogram, and cross-band envelope correlation matrix for each audio clip. Our resulting background textures reproduce closely the features of the input recordings. The sounds are from Font et al. [2013].
Fig. 12. The RMS errors from comparing our background texture synthesis and the McDermott et al. [2011] to the input sound texture over the texture statistics. We evaluate the texture statistics with the mean of the cochlear envelopes of each frequency band (Envelopes mean), the cross-band enve- lope correlation matrix (C), the modulation power (Mod. power) and two types of modulation correlations (C1: the same modulation frequency but different acoustic frequencies; C2: the same acoustic frequency but different modulation frequencies). The bar chart shows that our resulting background textures have a smaller RMS error from the input recordings.
Fig. 12. The RMS errors from comparing our background texture synthesis and the McDermott et al. [2011] to the input sound texture over the texture statistics. We evaluate the texture statistics with the mean of the cochlear envelopes of each frequency band (Envelopes mean), the cross-band enve- lope correlation matrix (C), the modulation power (Mod. power) and two types of modulation correlations (C1: the same modulation frequency but different acoustic frequencies; C2: the same acoustic frequency but different modulation frequencies). The bar chart shows that our resulting background textures have a smaller RMS error from the input recordings.
Fig. 13. Comparison of our HRTF construction with and without the hierar- chical grid with respect to the size of a sphere or plane. The two experiments indicate our hierarchical grid method improves performance and is slightly dependent on the size of the sound source.
Fig. 13. Comparison of our HRTF construction with and without the hierar- chical grid with respect to the size of a sphere or plane. The two experiments indicate our hierarchical grid method improves performance and is slightly dependent on the size of the sound source.
RAS Eg REESE Se re Fig. 14. Comparison of our HRTF construction to the SH method based on Schissler et al. [2016]. (a) For spheres, our method is slightly slower than the SH method which has only analytical projection. (b) For planes, our method is faster than the SH method using Monte Carlo projection with both 1000 and 10, 000 rays.
RAS Eg REESE Se re Fig. 14. Comparison of our HRTF construction to the SH method based on Schissler et al. [2016]. (a) For spheres, our method is slightly slower than the SH method which has only analytical projection. (b) For planes, our method is faster than the SH method using Monte Carlo projection with both 1000 and 10, 000 rays.

Related papers

Design of a Generative Model for Soundscape Creation
Nahla Mattar

2005

This paper describes the design and preliminary implementation, of a generative model for dynamic, real time soundscape creation. Our model is based on the work of the Acoustic Ecology community and provides a framework for the automated creation of compelling sonic environments that are both real and imagined. We outline extensions to the model that include interaction paradigms, context modeling, sound acquisition, and sound synthesis. Our work is flexible and extensible to a variety of different applications.

View PDFchevron_right
AUDIO TEXTURES
Remailer Punk

In this paper, we introduce a new audio medium, called audio texture, as a means of synthesizing long audio stream according to a given short example audio clip. The example clip is analyzed, and basic building patterns are extracted. Then an audio stream of arbitrary length is synthesized using a sequence of extracted building patterns. The patterns can be varied in the synthesis process to add variations to the generated sound. Audio textures are useful in applications such as background music, lullabies, game music, and screen saver sounds. A method is proposed for implementing audio textures. Preliminary results of audio textures are provided at our website for evaluation.

View PDFchevron_right
AcouRe: Streamlining the generation of immersive soundscapes from monophonic sources
Eduardo M C Magalhaes

The growing number of curated and crowdsourced online sound databases has been transforming several branches of music from production to consumption. In this context, the operational basis of sound design practice has been gradually shifting to incorporate repurposing strategies. One of the most pressing matters in this new context is the retrieval and transformation of audio content to meet the creative endeavours of the sound designer. The latter is the focus of our work, which aims to streamline the sound design process in the context of post-production, particularly when adopting large amounts of poorly annotated data. Different methodologies at the crossroads of machine listening and learning (namely, pattern recognition) are adopted to propose the automatic identification of sound events components from soundscape recordings to be repurposed in the temporal and spatial domains. A prototype interface, named AcouRe, implements the proposed methodologies. A heuristic evaluation to assess the usability of AcouRe was conducted by two expert sound designers.

View PDFchevron_right
GrainSynth: A Generative Synthesis Tool Based on Spatial Interpretations of Sound Samples
João Tenera

2021

This paper proposes a generative design approach for the creative exploration of dynamic soundscapes that can be used to generate compelling and immersive sound environments. A granular synthesis tool is considered based on the perceptual self-organization of sound samples by utilizing the t-Stochastic Neighboring Embedded algorithm (t-SNE) for the spatial mapping of sonic grains into a 2D space. The proposed system was able to relate the visual stimuli with the sonic responses in the context of the generic gestalt principles of visual perception. According to user evaluation, the application operated intuitively and also revealed the potential for creative expressiveness both from the user's perspective and as a standalone, generative synthesizer.

View PDFchevron_right
Mapping Sound to Image in Interactive Multimedia Art
Jonathan Berger

We describe an approach and framework for integrating synthesis and processing of digital image and sound. The approach is based on creating an analogy between sound and image by introducing the notion of the soxel, a representation analogous to the pixel. We describe some simple mappings between two domains that map pitch, time, spatial coordinates and timbre to bitmap mode, gray-scale mode, RGB color mode, and layer mode for images. The framework described, sonART, is a powerful multimedia application for integration of image and sound processing with flexible network communication.

View PDFchevron_right
A New Paradigm for Sound Design
Perry Cook

2006

A sound scene can be defined as any "environmental" sound that has a consistent background texture, with one or more potentially recurring foreground events. We describe a data-driven framework for analyzing, transforming, and synthesizing high-quality sound scenes, with flexible control over the components of the synthesized sound. Given one or more sound scenes, we provide welldefined means to: (1) identify points of interest in the sound and extract them into reusable templates, (2) transform sound components independently of the background or other events, (3) continually re-synthesize the background texture in a perceptually convincing manner, and (4) controllably place event templates over the background, varying key parameters such as density, periodicity, relative loudness, and spatial positioning. Contributions include: techniques and paradigms for template selection and extraction, independent sound transformation and flexible re-synthesis; extensions to a wavelet-based background analysis/synthesis; and user interfaces to facilitate the various phases. Given this framework, it is possible to completely transform an existing sound scene, dynamically generate sound scenes of unlimited length, and construct new sound scenes by combining elements from different sound scenes. URL:

View PDFchevron_right
SoundRiver: Semantically-Rich Sound Illustration
Rita Borgo

Computer Graphics Forum, 2010

Sound is an integral part of most movies and videos. In many situations, viewers of a video are unable to hear the sound track, for example, when watching it in a fast forward mode, viewing it by hearing-impaired viewers or when the plot is given as a storyboard. In this paper, we present an automated visualization solution to such problems. The system first detects the common components (such as music, speech, rain, explosions, and so on) from a sound track, then maps them to a collection of programmable visual metaphors, and generates a composite visualization. This form of sound visualization, which is referred to as SoundRiver, can be also used to augment various forms of video abstraction and annotated key frames and to enhance graphical user interfaces for video handling software. The SoundRiver conveys more semantic information to the viewer than traditional graphical representations of sound illustration, such as phonoautographs, spectrograms or artistic audiovisual animations.

View PDFchevron_right
Consideration of Effective Acoustic Rendering of Spatialized Ambient Sound
Maori Kobayashi

Interdisciplinary Information Sciences, 2012

Three-dimensional sound auralization systems have been developed actively in the last few decades. Such systems are called virtual auditory displays (VADs). In conventional VADs based on head-related transfer functions (HRTFs), a sound source position alone is rendered by disregarding other acoustical phenomena. However, because various sounds surround us in our daily life, we usually hear not only a targeted direct sound but also ambient sounds in an actual sound space. A lack of ambient sound often engenders an unnatural perception of the virtual auditory space presented by VAD based on HRTFs. Therefore, ambient sounds should be included in the VAD system auralization. We investigated an effective rendering method of ambient sound using ordinary colored noise. Furthermore, using subjective evaluations, we discuss the relation between the realism of sound space with ambient sounds and a listener's head movement.

View PDFchevron_right
Spatial Sound Rendering – A Survey
Eftychia Lakka

International Journal of Interactive Multimedia and Artificial Intelligence

Simulating propagation of sound and audio rendering can improve the sense of realism and the immersion both in complex acoustic environments and dynamic virtual scenes. In studies of sound auralization, the focus has always been on room acoustics modeling, but most of the same methods are also applicable in the construction of virtual environments such as those developed to facilitate computer gaming, cognitive research, and simulated training scenarios. This paper is a review of state-of-the-art techniques that are based on acoustic principles that apply not only to real rooms but also in 3D virtual environments. The paper also highlights the need to expand the field of immersive sound in a web based browsing environment, because, despite the interest and many benefits, few developments seem to have taken place within this context. Moreover, the paper includes a list of the most effective algorithms used for modelling spatial sound propagation and reports their advantages and disadvantages. Finally, the paper emphasizes in the evaluation of these proposed works.

View PDFchevron_right
Progressive perceptual audio rendering of complex scenes
George Drettakis

Proceedings of the 2007 symposium on Interactive 3D graphics and games - I3D '07, 2007

: Left: A scene with 1815 mobile sound sources. Audio is rendered in realtime with our progressive lossy processing technique using 15% of the frequency coefficients and with an average of 12 clusters for 3D audio processing. Degradations compared to the reference solution are minimal. Right: a scene with 1004 mobile sound sources, running with 25% of the frequency coefficients and 12 clusters.

View PDFchevron_right

References (54)

  1. V Ralph Algazi, Richard O Duda, Dennis M Thompson, and Carlos Avendano. 2001. The CIPIC HRTF database. In Applications of Signal Processing to Audio and Acoustics. IEEE, 2001 IEEE Workshop, 99-102.
  2. Durand R Begault and Leonard J Trejo. 2000. 3-D sound for virtual reality and multi- media. (2000).
  3. Juan Pablo Bello, Laurent Daudet, Samer Abdallah, Chris Duxbury, Mike Davies, and Mark B Sandler. 2005. A tutorial on onset detection in music signals. IEEE Transac- tions on speech and audio processing 13, 5 (2005), 1035-1047.
  4. Steven Boll. 1979. Suppression of acoustic noise in speech using spectral subtraction. IEEE Transactions on acoustics, speech, and signal processing 27, 2 (1979), 113-120.
  5. Joan Bruna and Stéphane Mallat. 2013. Audio texture synthesis with scattering moments. arXiv preprint arXiv:1311.0407 (2013).
  6. Nicholas Bryan and Gautham Mysore. 2013. An efficient posterior regularized latent variable model for interactive sound source separation. In International Conference on Machine Learning. 208-216.
  7. Chunxiao Cao, Zhong Ren, Carl Schissler, Dinesh Manocha, and Kun Zhou. 2016. Interactive sound propagation with bidirectional path tracing. ACM Transactions on Graphics (TOG) 35, 6 (2016), 1-11.
  8. Jeffrey N Chadwick and Doug L James. 2011. Animating fire with sound. In ACM Transactions on Graphics (TOG), Vol. 30. ACM, 84.
  9. Abe Davis and Maneesh Agrawala. 2018. Visual Rhythm and Beat. ACM Trans. Graph. 37, 4 (2018), 122-1.
  10. Antonio Di Crescenzo and Maria Longobardi. 2009. On cumulative entropies. Journal of Statistical Planning and Inference 139, 12 (2009), 4072-4087.
  11. Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. 2017. CARLA: An open urban driving simulator. arXiv preprint arXiv:1711.03938 (2017).
  12. Michael J Evans, James AS Angus, and Anthony I Tew. 1998. Analyzing head-related transfer function measurements using surface spherical harmonics. The Journal of the Acoustical Society of America 104, 4 (1998), 2400-2411.
  13. Raphael A. Finkel and Jon Louis Bentley. 1974. Quad trees a data structure for retrieval on composite keys. Acta informatica 4, 1 (1974), 1-9.
  14. Frederic Font, Gerard Roma, and Xavier Serra. 2013. Freesound technical demo. In Proceedings of the 21st ACM international conference on Multimedia. 411-412.
  15. Fabio P Freeland, Luiz WP Biscainho, and Paulo SR Diniz. 2002. Efficient HRTF interpola- tion in 3D moving sound. In Audio Engineering Society Conference: 22nd International Conference: Virtual, Synthetic, and Entertainment Audio. Audio Engineering Society.
  16. Hannes Gamper. 2013. Head-related transfer function interpolation in azimuth, ele- vation, and distance. The Journal of the Acoustical Society of America 134, 6 (2013), EL547-EL553.
  17. Aki Härmä, Julia Jakka, Miikka Tikander, Matti Karjalainen, Tapio Lokki, Jarmo Hi- ipakka, and Gaëtan Lorho. 2004. Augmented reality audio for mobile and wearable appliances. Journal of the Audio Engineering Society 52, 6 (2004), 618-639.
  18. Toni Heittola, Annamaria Mesaros, Dani Korpi, Antti Eronen, and Tuomas Virtanen. 2014. Method for creating location-specific audio textures. EURASIP Journal on Audio, Speech, and Music Processing 2014, 1 (2014), 9.
  19. Alexander JE Kell and Josh H McDermott. 2019. Invariance to background noise as a signature of non-primary auditory cortex. Nature communications 10, 1 (2019), 1-11.
  20. Vivek Kwatra, Irfan Essa, Aaron Bobick, and Nipun Kwatra. 2005. Texture optimization for example-based synthesis. In ACM SIGGRAPH 2005 Papers. 795-802.
  21. Vivek Kwatra, Arno Schödl, Irfan Essa, Greg Turk, and Aaron Bobick. 2003. Graphcut textures: image and video synthesis using graph cuts. ACM Transactions on Graphics (ToG) 22, 3 (2003), 277-286.
  22. Wei-Hsiang Liao, Axel Roebel, and Alvin Su. 2013. On the modeling of sound textures based on the STFT representation. In Proc. of the 16th Int. Conference on Digital Audio Effects (DAFx-13). 33.
  23. Shiguang Liu, Haonan Cheng, and Yiying Tong. 2019. Physically-based statistical simulation of rain sound. ACM Transactions on Graphics (TOG) 38, 4 (2019), 123.
  24. Josh H McDermott and Eero P Simoncelli. 2011. Sound texture perception via statistics of the auditory periphery: evidence from sound synthesis. Neuron 71, 5 (2011), 926-940.
  25. Brian McFee, Justin Salamon, and Juan Pablo Bello. 2018. Adaptive pooling operators for weakly labeled sound event detection. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP) 26, 11 (2018), 2180-2193.
  26. Ian McLoughlin, Haomin Zhang, Zhipeng Xie, Yan Song, and Wei Xiao. 2015. Robust sound event classification using deep neural networks. IEEE/ACM Transactions on Audio, Speech, and Language Processing 23, 3 (2015), 540-552.
  27. Lindasalwa Muda, Mumtaj Begam, and Irraivan Elamvazuthi. 2010. Voice recognition algorithms using mel frequency cepstral coefficient (MFCC) and dynamic time warping (DTW) techniques. arXiv preprint arXiv:1003.4083 (2010).
  28. Sean O'Leary and Axel Roebel. 2014. A two level montage approach to sound texture synthesis with treatment of unique events.. In DAFx. 1-1.
  29. Seán O'Leary and Axel Röbel. 2016. A montage approach to sound texture synthesis. IEEE/ACM Transactions on Audio, Speech, and Language Processing 24, 6 (2016), 1094-1105.
  30. Ashish Panda and Thambipillai Srikanthan. 2011. Psychoacoustic model compensation for robust speaker verification in environmental noise. IEEE transactions on audio, speech, and language processing 20, 3 (2011), 945-953.
  31. David R Perrott and Kourosh Saberi. 1990. Minimum audible angle thresholds for sources varying in both elevation and azimuth. The Journal of the Acoustical Society of America 87, 4 (1990), 1728-1731.
  32. Emil Praun, Adam Finkelstein, and Hugues Hoppe. 2000. Lapped textures. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques. 465- 470.
  33. Lawrence R Rabiner and Ronald W Schafer. 2011. Theory and applications of digital speech processing. Vol. 64. Pearson Upper Saddle River, NJ.
  34. Boaz Rafaely and Amir Avni. 2010. Interaural cross correlation in a sound field repre- sented by spherical harmonics. The Journal of the Acoustical Society of America 127, 2 (2010), 823-828.
  35. Nikunj Raghuvanshi, Rahul Narain, and Ming C Lin. 2009. Efficient and accurate sound propagation using adaptive rectangular decomposition. IEEE Transactions on Visualization and Computer Graphics 15, 5 (2009), 789-801.
  36. Nikunj Raghuvanshi and John Snyder. 2018. Parametric directional coding for pre- computed sound propagation. ACM Transactions on Graphics (TOG) 37, 4 (2018), 108. Curtis Roads. 1988. Introduction to granular synthesis. Computer Music Journal 12, 2 (1988), 11-13.
  37. Griffin D Romigh, Douglas S Brungart, Richard M Stern, and Brian D Simpson. 2015. Efficient real spherical harmonic representation of head-related transfer functions. IEEE Journal of Selected Topics in Signal Processing 9, 5 (2015), 921-930.
  38. Nicolas Saint-Arnaud and Kris Popat. 1995. Analysis and synthesis of sound textures. In in Readings in Computational Auditory Scene Analysis. Citeseer.
  39. Carl Schissler, Ravish Mehra, and Dinesh Manocha. 2014. High-order diffraction and diffuse reflections for interactive sound propagation in large environments. ACM Transactions on Graphics (TOG) 33, 4 (2014), 1-12.
  40. Carl Schissler, Aaron Nicholls, and Ravish Mehra. 2016. Efficient HRTF-based spa- tial audio for area and volumetric sources. IEEE transactions on visualization and computer graphics 22, 4 (2016), 1356-1366.
  41. Diemo Schwarz. 2011. State of the art in sound texture synthesis. In Digital Audio Effects (DAFx). 221-232.
  42. Diemo Schwarz and Baptiste Caramiaux. 2013. Interactive sound texture synthesis through semi-automatic user annotations. In International Symposium on Computer Music Multidisciplinary Research. Springer, 372-392.
  43. Mincheol Shin, Stephen W Song, Se Jung Kim, and Frank Biocca. 2019. The effects of 3D sound in a 360-degree live concert video on social presence, parasocial interaction, enjoyment, and intent of financial supportive action. International Journal of Human- Computer Studies 126 (2019), 81-93.
  44. Paris Smaragdis, Bhiksha Raj, and Madhusudana Shashanka. 2006. A probabilistic latent variable model for acoustic modeling. (2006).
  45. Martin Spiertz and Volker Gnann. 2009. Source-filter based clustering for monaural blind source separation. In Proceedings of the 12th International Conference on Digital Audio Effects.
  46. Yapeng Tian, Chenliang Xu, and Dingzeyu Li. 2019. Deep Audio Prior. ArXiv abs/1912.10292 (2019).
  47. Andries Van Der Merwe and Walter Schulze. 2010. Music generation with markov models. IEEE MultiMedia 18, 3 (2010), 78-85.
  48. Charles Verron, Mitsuko Aramaki, Richard Kronland-Martinet, and Grégory Pallone. 2009. Spatialized synthesis of noisy environmental sounds. In Auditory Display. Springer, 392-407.
  49. Jui-Hsien Wang, Ante Qu, Timothy R Langlois, and Doug L James. 2018. Toward wave-based sound synthesis for computer animation. ACM Trans. Graph. 37, 4 (2018), 109-1.
  50. Stephan Wenger and Marcus Magnor. 2011. Constrained example-based audio synthesis. In 2011 IEEE International Conference on Multimedia and Expo. IEEE, 1-6.
  51. Zechen Zhang, Nikunj Raghuvanshi, John Snyder, and Steve Marschner. 2018. Ambient sound propagation. ACM Transactions on Graphics (TOG) 37, 6 (2018), 1-10.
  52. Zechen Zhang, Nikunj Raghuvanshi, John Snyder, and Steve Marschner. 2019. Acoustic texture rendering for extended sources in complex scenes. ACM Transactions on Graphics (TOG) 38, 6 (2019), 1-9.
  53. Changxi Zheng and Doug L James. 2009. Harmonic fluids. In ACM Transactions on Graphics (TOG), Vol. 28. ACM, 37.
  54. Xinglei Zhu and Lonce Wyse. 2004. Sound texture modeling and time-frequency LPC. In Proceedings of the 7th international conference on digital audio effects DAFX, Vol. 4.

Related papers

State of the Art in Sound Texture Synthesis
Diemo Schwarz

2011

The synthesis of sound textures, such as rain, wind, or crowds, is an important application for cinema, multimedia creation, games and installations. However, despite the clearly defined requirments of naturalness and flexibility, no automatic method has yet found widespread use. After clarifying the definition, terminology, and usages of sound texture synthesis, we will give an overview of the many existing methods and approaches, and the few available software implementations, and classify them by the synthesis model they are based on, such as subtractive or additive synthesis, granular synthesis, corpus-based concatenative synthesis, wavelets, or physical modeling. Additionally, an overview is given over analysis methods used for sound texture synthesis, such as segmentation, statistical modeling, timbral analysis, and modeling of transitions.

View PDFchevron_right
Sound rendering
James Hahn

ACM SIGGRAPH Computer Graphics, 1992

We present a general methodology to produce synchronized soundtracks for animations. A sound world is modeled by associating a characteristic sound for each object in a scene. These sounds can be generated from a behavioral or physically-based simulation. Collision sounds can be computed from vibrational response of elastic bodies to the collision impulse. Alternatively, stereotypic recorded sound effects can be associated with each interaction of objects. Sounds may also be generated procedurally, The sound world is described with a sound event file, and is rendered in two passes. First the propagation paths from 3D objects to each microphone are analyzed and used to calculate sound transformations according to the acoustic environment. These effects are convolutions, encoded into two essential parameters, delay and attenuation of each sound. Timeciependency of these two parameters is represented with key frames, thus being completely independent of the original 3D animation script. In the second pass the sounds associated with objects are instantiated, modulated by interpolated key parameters, and summed up to the final soundtrack. The advantage of a modular architecture is that the same methods can be used for all types of animations, keyframed, physically-based and behavioral. We also discuss the differences of sound and light, and the remarkable similarities in their rendering processes.

View PDFchevron_right
The SoundScape Renderer: A unified spatial audio reproduction framework for arbitrary rendering methods
Matthias Geier, Sascha Spors

124th AES Convention, 2008

The papers at this Convention have been selected on the basis of a submitted abstract and extended precis that have been peer reviewed by at least two qualified anonymous reviewers. This convention paper has been reproduced from the author's advance manuscript, without editing, ...

View PDFchevron_right
SoundScapeGenerator: soundscape modelling and simulation
Marinos Koutsomichalis, Andrea Valle

Atti del XX CIM ‐ Colloquio di Informatica Musicale ‐ MUSICHE LIQUIDE, 2014

This paper describes SoundScapeGenerator, a generic system for modelling and simulating soundscapes both in real and non-real time. SoundsScapeGenerator features algorithms for three-dimensional sound-localisation and is able to model complex spaces. The system relies on abstract rule descriptions (generators) to simulate deterministic, “cartoonified” (loosely modelled) sequencing of sound events. It further supports virtual P.O.H. (point of hearing) and is able to render the final output in various channel configurations. Based on generative algorithms, the SoundScapeGenerator implementation allows real-time user interaction with ever-changing soundscapes, and it can easily communicate with other applications and devices. Finally, we introduce SoundScapeComposer, a higher level module developed for the SoDA project (dedicated to soundscape automatic generation), that has been built on top of SoundScapeGenerator, hiding most of the latter's details to the user.

View PDFchevron_right
THE SOUNDSCAPE RENDERER: A VERSATILE SOFTWARE FRAMEWORK FOR SPATIAL AUDIO REPRODUCTION
Katharina Bredies

We present a new software system for spatial audio repro- duction called SoundScape Renderer. It is currently used for rendering 2-dimensional virtual acoustic scenes using either Wave Field Synthesis, binaural rendering or Vector Based Amplitude Panning. A further extension to higher order Am- bisonics is planned. However, in this paper we will focus on the WFS functionality. We also describe the physical setup of the WFS system in- stalled at the Usability Laboratory of Deutsche Telekom Lab- oratories, the natural habitat of the SoundScape Renderer. Finally, we present a working draft for an XML file for- mat enabling the exchange of acoustic scenes between differ- ent software systems. This Audio Scene Description Format (ASDF) is entirely open and we encourage everyone to par- ticipate in its further definition.

View PDFchevron_right

Related topics

  • Information Systems
  • Computer Science
  • Acoustics
  • Texture Synthesis
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved