Academia.eduAcademia.edu

Outline

Title
Abstract
Figures
Introduction
Conclusion
Acknowledgements
References
All Topics
Mathematics

Kernel techniques for generalized audio crossfades

Profile image of William SetharesWilliam Sethares

2015, Cogent Mathematics

https://doi.org/10.1080/23311835.2015.1102116
visibility

…

description

22 pages

link

1 file

Abstract

This paper explores a variety of density and kernel-based techniques that can smoothly connect (crossfade or "morph" between) two functions. When the functions represent audio spectra, this provides a concrete way of adjusting the partials of a sound while smoothly interpolating between existing sounds. The approach can be applied to both interpolation-crossfades (where the crossfade connects two different sounds over a specified duration) and to repetitive-crossfades (where a series of sounds are generated, each containing progressively more features of one sound and fewer of the other). The interpolation surface can be thought of as the two dimensions (time and frequency) of a spectrogram, and the kernels can be chosen so as to constrain the surface in a number of desirable ways. When successful, the timbre of the sounds is changed dynamically in a plausible way. A series of sound examples demonstrate the strengths and weaknesses of the approach.

Figures (5)
Figure 1: Audio crossfades generate sounds that change smoothly between a source and a destination sound. In interpolation crossfades (a), the sound be- gins as A and over time smoothly becomes like B. The total duration of the output sound is independent of the duration of A and B and the cross only de- pends on the sound in the starting and ending frames. The overall effect is one of stretching time under the constraint that the sound must emerge continuously from A and merge continuously into B. In repetitive crossfades (b), a series of intermediate sounds M; merge aspects of A and B, analogous to the intermedi- ary photographs of an image morph that merges various aspects of the starting and ending photographs. The duration of each output sound /V/; is equal to the common duration of A and B. Thus interpolation crosses begin as one sound and end as another while in a repetitive cross, each V/; contains features of both of the original sounds. For instance, an interpolation crossfade might start with the attack portion of a cymbal and end with the final moments of a lion’s roar. The interpolation crossfade is the transition that occurs over a user specified time. In contrast, each intermediate sound in a repetitive crossfade merges aspects of both the complete lion sound (from start to end) with those of the complete cymbal (from attack through decay).
Figure 1: Audio crossfades generate sounds that change smoothly between a source and a destination sound. In interpolation crossfades (a), the sound be- gins as A and over time smoothly becomes like B. The total duration of the output sound is independent of the duration of A and B and the cross only de- pends on the sound in the starting and ending frames. The overall effect is one of stretching time under the constraint that the sound must emerge continuously from A and merge continuously into B. In repetitive crossfades (b), a series of intermediate sounds M; merge aspects of A and B, analogous to the intermedi- ary photographs of an image morph that merges various aspects of the starting and ending photographs. The duration of each output sound /V/; is equal to the common duration of A and B. Thus interpolation crosses begin as one sound and end as another while in a repetitive cross, each V/; contains features of both of the original sounds. For instance, an interpolation crossfade might start with the attack portion of a cymbal and end with the final moments of a lion’s roar. The interpolation crossfade is the transition that occurs over a user specified time. In contrast, each intermediate sound in a repetitive crossfade merges aspects of both the complete lion sound (from start to end) with those of the complete cymbal (from attack through decay).
Figure 2: A crossfade surface can be defined by Laplace’s equation V7u(x, y) = 0 with boundary conditions given by the spectra of two sounds A and B. The x-axis (representing time) proceeds from time 0 to time t* while the y-axis (representing frequency) covers the range from DC (at 0) to the Nyquist rate (at 1). The surface is formally analogous to a spectrogram and can be inverted back into the time domain using any of a variety of standard techniques. film is, in essence, reinterpreted as a spectrogram. frame is dipped into a pool of soapy water and carefully retracted, a smooth sheet forms that is characterized as the surface that minimizes the surface energy where the height of the sheet at each point is u(x, y). Mathematically, this can be stated as the PDE (1) with the specified boundary conditions. Reinterpreting the contour of the soap film (i.e., the field values) as sound provides the audio output, which can be heard to smoothly interpolate from the left hand spectrum to the right hand spectrum. This views the crossfade function as the solution to a boundary value problem over a two-dimensional domain defined by the spectrum of the sound in the y dimension and the duration of the crossfade in the x direction. The soapy film is, in essence, reinterpreted as a spectrogram.
Figure 2: A crossfade surface can be defined by Laplace’s equation V7u(x, y) = 0 with boundary conditions given by the spectra of two sounds A and B. The x-axis (representing time) proceeds from time 0 to time t* while the y-axis (representing frequency) covers the range from DC (at 0) to the Nyquist rate (at 1). The surface is formally analogous to a spectrogram and can be inverted back into the time domain using any of a variety of standard techniques. film is, in essence, reinterpreted as a spectrogram. frame is dipped into a pool of soapy water and carefully retracted, a smooth sheet forms that is characterized as the surface that minimizes the surface energy where the height of the sheet at each point is u(x, y). Mathematically, this can be stated as the PDE (1) with the specified boundary conditions. Reinterpreting the contour of the soap film (i.e., the field values) as sound provides the audio output, which can be heard to smoothly interpolate from the left hand spectrum to the right hand spectrum. This views the crossfade function as the solution to a boundary value problem over a two-dimensional domain defined by the spectrum of the sound in the y dimension and the duration of the crossfade in the x direction. The soapy film is, in essence, reinterpreted as a spectrogram.
Figure 3: Sinusoids of frequencies w;, = 5 and wy, = 12 are crossed with frequencies Wr, = 6 and wr, = 11 using the Poisson kernel and three different G(«) functions (see text for details). Though the ridges connecting the nearby frequencies appear in all three figures, the drop in (a) is likely to be heard as a drop in volume over the course of the first half of the crossfade. This is plotted in Fig. 3(a). The boundaries at the left and right show the two sinusoids (as delta functions at their respective frequencies) while the surface gradually descends to the middle where they meet. Observe that there are two shapes that connect the nearby frequencies w,, to wr, and wy, to wr,. These are local maxima (in the y direction) which form a connected set as x varies over its range; call these ridges. Observe that there is a significant loss of height in the ridges of Fig. 3(a). Since the magnitude of the surface corresponds to the amplitude of the spectral components, this may be perceptible as a drop in the volume towards the middle of the crossfade region.
Figure 3: Sinusoids of frequencies w;, = 5 and wy, = 12 are crossed with frequencies Wr, = 6 and wr, = 11 using the Poisson kernel and three different G(«) functions (see text for details). Though the ridges connecting the nearby frequencies appear in all three figures, the drop in (a) is likely to be heard as a drop in volume over the course of the first half of the crossfade. This is plotted in Fig. 3(a). The boundaries at the left and right show the two sinusoids (as delta functions at their respective frequencies) while the surface gradually descends to the middle where they meet. Observe that there are two shapes that connect the nearby frequencies w,, to wr, and wy, to wr,. These are local maxima (in the y direction) which form a connected set as x varies over its range; call these ridges. Observe that there is a significant loss of height in the ridges of Fig. 3(a). Since the magnitude of the surface corresponds to the amplitude of the spectral components, this may be perceptible as a drop in the volume towards the middle of the crossfade region.
Figure 4: The ridges in the crossfade surface on the left are equally wide irrespec- tive of the absolute frequency. In some situations, it may be advantageous to allow the width of the ridges to become wider at higher frequencies, as shown on the right. This can be accomplished by defining the kernels as suggested in Example 3.
Figure 4: The ridges in the crossfade surface on the left are equally wide irrespec- tive of the absolute frequency. In some situations, it may be advantageous to allow the width of the ridges to become wider at higher frequencies, as shown on the right. This can be accomplished by defining the kernels as suggested in Example 3.

Related papers

Musical Instrument Sound Morphing Guided by Perceptually Motivated Features
Marcelo Caetano

IEEE Transactions on Audio, Speech, and Language Processing, 2000

Sound morphing is a transformation that gradually blurs the distinction between the source and target sounds. For musical instrument sounds, the morph must operate across timbre dimensions to create the auditory illusion of hybrid musical instruments. The ultimate goal of sound morphing is to perform perceptually linear transitions, which requires an appropriate model to represent the sounds being morphed and an interpolation function to obtain intermediate sounds. Typically, morphing techniques directly interpolate the parameters of the sound model without considering the perceptual impact or evaluating the results. Perceptual evaluations are cumbersome and not always conclusive. In this work, we seek parameters of a sound model that favor linear variation of perceptually motivated temporal and spectral features used to guide the morph towards more perceptually linear results. The requirement of linear variation of feature values gives rise to objective evaluation criteria for sound morphing. We investigate several spectral envelope morphing techniques to determine which spectral representation renders the most linear transformation in the spectral shape feature domain. We found that interpolation of line spectral frequencies gives the most linear spectral envelope morphs. Analogously, we study temporal envelope morphing techniques and we concluded that interpolation of cepstral coefficients results in the most linear temporal envelope morph.

View PDFchevron_right
Automatic Audio Morphing On Detached Sound Waveforms
Amarjot Singh

This paper describes the techniques to morph between the two portions of sound originated from a common source, broken because of by some reason. We make use of morphing technique to join the sound where the pitch of the sound slowly changes from one broken part to another broken part by slowly changing the pitch information also covering the unvoiced region of the music. Spectral shapes are encoded on the multidimensional space while pitch on orthogonal axes of it. After matching components of the sound, a morph smoothly interpolates the amplitudes to describe a new sound in the same perceptual space. Finally, by inverting the representation, sound is produced. According to the literature survey, this is the first paper which explains the step by step evolution of one sound resulting into other sound across all states of intermediate steps along with the step by step runtime analysis. This will help researchers in future to improve the morphing methodology by looking carefully into the intermediate steps which decides the overall results. Here, we key out representations for morphing, techniques for matching, and interpolation algorithms and morphing each sound component. Spectral images of complete morph spectrogram are shown in the end.

View PDFchevron_right
Automatic audio morphing
Michele Covell

International Conference on Acoustics, Speech, and Signal Processing, 1996

This paper describes techniques to automatically morph from one sound to another. Audio morphing is accomplished by representing the sound in a multi-dimensional space that is warped or modified to produce a desired result. The multi-dimensional space encodes the spectral shape and pitch on orthogonal axes. After matching components of the sound, a morph smoothly interpolates the amplitudes to describe

View PDFchevron_right
Sound morphing strategies based on alterations of time-frequency representations by Gabor multipliers
Bruno Torresani

Sounds morphing is an important topic in signal processing of musical sounds and covers a wide variety of techniques whose aim is to "interpolate" between two sound signals. We present here an approach based on the alteration of time-frequency representation. Time-frequency analysis is a classical tool in sounds analysis/synthesis. A time-frequency filter can be well-defined as a diagonal signal operator in a Gabor representation of sounds. Processing can be performed by multiplying a time-frequency representation with such a time-frequency filter, called a Gabor mask. After estimating such a Gabor mask between two sounds, we explore strategies to parametrize it for static morphing between two sounds. We then compare such an approach with standard and non standard approaches of morphing as different kind of sounds combination, notably classical means in the time-frequency domain.

View PDFchevron_right
Independent manipulation of high-level spectral envelope shape features for sound morphing by means of evolutionary computation
Marcelo Caetano

The aim of sound morphing is to obtain a sound that falls perceptually between two (or more) sounds. Ideally, we want to morph perceptually relevant features of sounds and be able to independently manipulate them. In this work we present a method to obtain perceptually intermediate spectral envelopes guided by highlevel spectral shape descriptors and a technique that employs evolutionary computation to independently manipulate the timbral features captured by the descriptors. High-level descriptors are measures of the acoustic correlates of salient timbre dimensions derived from perceptual studies, such that the manipulation of the descriptors corresponds to potentially interesting timbral variations.

View PDFchevron_right
Spectral analysis based synthesis and transformation of digital sound: the ATSH program
Oscar Pablo Di Liscia

2003

From the very beginning of computer music, the Fast Fourier Transform (FFT) has been considered a powerful technique that allowed the development of many valuable tools for research and transformation of digital sound. The reader may consult -among othersthe works by ( Moore,1990, 1978, Embree & Kimble, 1991, Moorer, 1978, and Wessel & Risset, 1985), in order to obtain the basic concepts needed for a detailed comprehension of the following discussion.

View PDFchevron_right
Concatenation and Stretch/squeeze of Musical instrumental sound using sound Morphing
Naotoshi Osaka

2005

Sound morphing is one of the successful synthesis technologies for recent computer music, and several studies have been reported on this subject. We have proposed a sound morphing algorithm based on a sinusoidal model. In this paper, a correspondence search algorithm is improved, which is core technology of morphing based on a sinusoidal model. Then wider applications of the algorithm are introduced. It can be used as concatenation of two sounds if morphing interval is as short as 100 msec. It is also used as stretch and squeeze of a sound, if the algorithm is applied to two copies from the same sample. Moreover, adding a musical expression to an instrumental sound are discussed.

View PDFchevron_right
Spectral Modeling of Musical Sounds
Xavier Serra

Fourth Meeting of the Fwo Research Society on Foundations of Music Research, 1998

View PDFchevron_right
A Multi-Point {2D} Interface: Audio-Rate Signals For Controlling Complex Multi-Parametric Sound Synthesis
Stuart James

2016

This paper documents a method of controlling complex sound synthesis processes such as granular synthesis, additive synthesis, timbre morphology, swarm-based spatialisation, spectral spatialisation, and timbre spatialisation via a multi-parametric 2D interface. This paper evaluates the use of audio-rate control signals for sound synthesis, and discussing approaches to de-interleaving, synchronization, and mapping. The paper also outlines a number of ways of extending the expressivity of such a control interface by coupling this with another 2D multi-parametric nodes interface and audio-rate 2D table lookup. The paper proceeds to review methods of navigating multi-parameter sets via interpolation and transformation. Some case studies are finally discussed in the paper. The author has used this method to control complex sound synthesis processes that require control data for more that a thousand parameters.

View PDFchevron_right
A Morphing Approach for Synthesizing Multichannel Recordings
Ching-Shun Lin

Conference Record of the Thirty-Ninth Asilomar Conference onSignals, Systems and Computers, 2005.

In this paper we propose a morphing approach that can synthesize microphone signals with the acoustical characteristics of specific instrument. The proposed method is composed of three stages. In the first step, the short-time Fourier transform is used to represent signals as a 3-D structure. The cosine similarity measure is then applied to this pre-processed time-frequency information for locating the most similar spectrogram in the matching database. Finally, the input spectrogram is adjusted according to the trend extracted from the pair-wise reference spectrograms by morphing algorithms. As an example of our method we present results on synthesis of microphone signals that capture a percussive instrument such as an orchestra tympani.

View PDFchevron_right

References (17)

  1. F. Boccardi and C. Drioli, "Sound Morphing With Gaussian Mixture Models," Proc. 4th COST G-6 Workshop on Digital Audio Effects, Limerick, Ireland, Dec. 2001.
  2. M Dolson, "The phase vocoder: a tutorial," Computer Music Journal, Spring, Vol. 10, No. 4, 14-27, 1986.
  3. T. Erbe, Soundhack Manual, Frog Peak Music, Lebanon, NH, 1994 (pp. 7- 40).
  4. E. Farnetani and D. Recasens, "Coarticulation and Connected Speech Pro- cesses," in Handbook of Phonetic Sciences, 2cnd Edition, Ed. W. J. Hardcas- tle, J. Laver, F. E. Gibbon, Blackwell Pubs. 2010 (pp. 316-352).
  5. K. Fitz, L. Haken, S. Lefvert, and M. O'Donnell, "Sound morphing using Loris and the reassigned bandwidth-enhanced additive sound model: Practice and applications," in International Computer Music Conference, Gotenborg, Sweden, 2002.
  6. K. E. Gustafson, Introduction to Partial Differential Equations and Hilbert Space Methods, John Wiley and Sons, Hoboken, NJ, 1980 (pp. 1-35).
  7. W. Hatch, High-Level Audio Morphing Strategies, MS Thesis, McGill Uni- versity, Aug. 2004.
  8. J. Laroche and M. Dolson, "Improved phase vocoder time-scale modification of audio," IEEE Trans. on Audio and Speech Processing, Vol. 7, No. 3, May 1999.
  9. R. J. McAulay and T. F. Quatieri, "Speech analysis/synthesis based on a si- nusoidal representation," IEEE Trans. on Acoustics, Speech, and Signal Pro- cessing ASSP-34(4), 744-754, 1986.
  10. F. Oberhettinger, Fourier Transforms of Distributions and Their Inverses Academic Press, New York, 1973 (pp. 15-17).
  11. A. V. Oppenheim and R. W. Schafer, Discrete-Time Signal Processing, 3rd Edition, Prentice-Hall, New Jersey, 2009 (pp. 730-742).
  12. L. Polansky, and M. McKinney, "Morphological mutation functions: ap- plications to motivic transformations and to a new class of cross-synthesis techniques," Proc. of the ICMC, Montreal, 1991.
  13. X. Serra, "Sound hybridization based on a deterministic plus stochastic de- composition model," in Proc. of the 1994 International Computer Music Con- ference, Aarhus, Denmark, 348351, 1994.
  14. W. A. Sethares, Rhythm and Transforms, Springer-Verlag, London, UK 2007 (pp. 111-145)
  15. W. A. Sethares, A. Milne, S. Tiedje , A. Prechtl and J. Plamondon, "Spectral tools for dynamic tonality and audio morphing," Computer Music Journal, Vol. 33, No. 2, Pages 71-84, Summer 2009.
  16. M. Slaney, M. Covell, and B. Lassiter, "Automatic audio morphing," Proc- ceedings of the 1996 International Conference on Acoustics, Speech, and Sig- nal Processing, Atlanta, GA, May 1996.
  17. E. Tellman, L. Haken, B. Holloway, "Timbre morphing of sounds with un- equal numbers of features" Journal of the Audio Engineering Society, Vol. 43, No. 9, 678-689, Sept. 1995.

Related papers

Laplacian Directed Audio Morphing
William Sethares

Inspired by a formal analogy between the two dimensional version of Laplace's partial differential equation and the two dimensions (time and frequency) of a spectrogram, this paper explores a variety of density and kernelbased techniques that can smoothly connect (morph between) two functions. When the functions represent audio spectra, this presents a solution to the problem of audio morphing, providing a concrete way of adjusting the overtones of a sound while smoothly interpolating between existing sounds. The approach can be applied to both interpolation morphing (where the morph connects two different sounds over some specified duration) and to repetitive morphing (where a series of sounds are generated, each containing progressively more features of one sound and fewer of the other). When successful, the timbre of the sounds is changed dynamically in a plausible way. A series of sound examples demonstrate the strengths and weaknesses of the approach. * Both sethares@ece.wisc.edu and bucklew@engr.wisc.edu are with the

View PDFchevron_right
SOUND MORPHING BY FEATURE INTERPOLATION
Marcelo Caetano

The goal of sound morphing by feature interpolation is to obtain sounds whose values of features are intermediate between those of the source and target sounds. In order to do this, we should be able to resynthesize sounds that present a set of predefined feature values, a notoriously difficult problem. In this work, we present morphing techniques to obtain hybrid musical instrument sounds whose feature values correspond as close as possible to the ideal interpolated values. When the features capture perceptually relevant information, the morphed sound whose features are interpolated is perceptually intermediate. The features we use are acoustic correlates of salient timbre dimensions derived from perceptual studies, such that sounds whose feature values are intermediate between two would be placed between them in the underlying timbre space. We measure the perceptual impact of the morphed sounds directly by the feature values, using them as an objective measure with which to evaluate the results. Thus we consider that the morphed sounds change perceptually linearly when the corresponding feature values vary linearly.

View PDFchevron_right
Spectral tools for Dynamic Tonality and audio morphing
Andrew Milne, William Sethares

The Spectral Toolbox is a suite of analysis–resynthesis programs that locate relevant partials of a sound and allow them to be resynthesized at specified frequencies. This enables a variety of routines including spectral mappings (changing all partials of a source sound to fixed destination frequencies), spectral morphing (continuously interpolating between the partials of a source sound and those of a destination sound), and what we call Dynamic Tonality (a novel way of organizing the relationship between a family of tunings and a set of related spectra). A complete application called the TransFormSynth provides a concrete implementation of Dynamic Tonality.

View PDFchevron_right
Spectral modeling for higher-level sound transformations
Xavier Serra

2001

Abstract When designing audio effects for music processing, we are always aiming at providing higherlevel representations that may somehow fill in the gap between the signal processing world and the end-user. Spectral models in general, and the Sinusoidal plus Residual model in particular, can sometimes offer ways to implement such schemes.

View PDFchevron_right
Morphing Sound in Real Time through the Timbre Tunnel
Johannes Kretz

2015

This paper presents the author’s experience in developing a software called “morph tunnel” for morphing sounds in real time in the frame of the MaxMSP environment, as well as its application for the composition and performance of a work, “timbre tunnel” for 6 musicians, dancer (optional) and live electronics. The main idea is to create a spectral – and temporal – “tunnel” through a method of pattern matching between overtones, where the user – or a dancer – can navigate in real time between spectra and time in a continuous way.

View PDFchevron_right

Related topics

  • Mathematics
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved