Real-time musical applications using the FFT-based resynthesis

2003

From the very beginning of computer music, the Fast Fourier Transform (FFT) has been considered a powerful technique that allowed the development of many valuable tools for research and transformation of digital sound. The reader may consult -among othersthe works by ( Moore,1990, 1978, Embree & Kimble, 1991, Moorer, 1978, and Wessel & Risset, 1985), in order to obtain the basic concepts needed for a detailed comprehension of the following discussion.

2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2012

The model used to represent musical instrument sounds plays a crucial role in the quality of sound transformations. Ideally, the representation should be compact and accurate, while its parameters should give flexibility to independently manipulate perceptually related features of the sounds. This work describes a source-filter model for musical instrument sounds based on the sinusoidal plus residual decomposition. The sinusoidal component is modeled as sinusoidal partial tracks (source) and a time-varying spectral envelope (filter), and the residual is represented as white noise (source) shaped by a time-varying spectral envelope (filter). This article presents estimation and representation techniques that give totally independent and intuitive control of the spectral envelope model and the frequencies of the partials to perform perceptually related sound transformations. The result of a listening test confirmed that, in general, the sounds resynthesized from the source-filter model are perceptually similar to the original recordings.

University of York, 1992

It has been said more than once that the advent of computer technology and its application to music has given the electroacoustic composer command over a vast universe of sound that could only be the dream of his predecessors. While, at least in theory, it is reasonable to assume the validity of this statement, it is also important to bear in mind that the same developments have given rise to new problems confronting anyone willing to take full advantage of the new developments. For instance, the psychoacoustic effects of sound perception within a musical context are immediately apparent and much more significant than when dealing with traditional instruments: A concept like timbre, which is reasonably defined within the orchestral palette, takes new significance as a time-varying, multidimensional entity, in which there is no one to one correspondence between the spectrum and timbral identity. Another well known difficulty is that involved in achieving sounds that are 'alive', rather than of neutral 'electronic' quality, experienced by anyone that has used a synthesizer, regardless of whether it is implemented as a piece of hardware or in a software package.

Proceedings of the Digital Music Research …, 2007

2014

Motion is a time-based event situated in threedimensional space. In the performance of most musical instruments, the musician associates physical motion with audible results. In some instruments, physical motion is used to change the timbre of the instrument. Digital techniques of analysis and re-synthesis have paved the way for the understanding of timbre as a distribution of energy among spectral frequencies in a three-dimensional space bounded by axes representing time, frequency and amplitude. In this paper, we present a musical performance system that explores the correlation between motion and sound, more specifically, timbre. A mobile device and its motion sensors function as a control element in the performance system that creates new timbres in real time.

2008

This paper presents an analysis-manipulation method that can generate musical instrument sounds with arbitrary pitches and durations from the sound of a given musical instrument (called seed) without distorting its timbral characteristics. Based on psychoacoustical knowledge of the auditory effects of timbres, we defined timbral features based on the spectrogram of the sound of a musical instrument as (i) the relative amplitudes of the harmonic peaks, (ii) the distribution of the inharmonic component, and (iii) temporal envelopes. First, to analyze the timbral features of a seed, it was separated into harmonic and inharmonic components using Itoyama's integrated model. For pitch manipulation, we took into account the pitch-dependency of features (i) and (ii). We predicted the values of each feature by using a cubic polynomial that approximated the distribution of these features over pitches. To manipulate duration, we focused on preserving feature (iii) in the attack and decay duration of a seed. Therefore, only steady durations were expanded or shrunk. In addition, we propose a method for reproducing the properties of vibrato. Experimental results demonstrated the quality of the synthesized sounds produced using our method. The spectral and MFCC distances between the synthesized sounds and actual sounds of 32 instruments were reduced by 64.70% and 32.31%, respectively.

This paper summarizes the authors experiences in developing tools for generating and performing sounds with computers using concepts, which strongly extend the traditional concept of " notes in a score ". The software environment KLANGPILOT allows to describe and formalize musical ideas in a language, which is both, close enough to traditional concepts to support an efficient creative process, and rich enough to allow the creation of (mainly synthetic) sounds with unusual, sophisticated spectral properties. Also some links between high level control of sound synthesis and computer aided composition are highlighted. Finally a way of playing back sounds is described, which provides a compromise between the safety of recorded tapes and the temporal flexibility of real time technology.

This paper presents an implementation of a near real-time timbre morphing signal processing system, designed to facilitate an element of 'liveness' and unpredictability in a musical installation. The timbre morpher is a hybrid analysis and synthesis technique based on Spectral Modeling Synthesis (an additive and subtractive modeling technique). The musical installation forms an interactive soundtrack in response to the series of Rosso Luana marble sculptures Shapes in the Clouds, I, II, III, IV & V by artist Peter Randall-Page, exhibited at the Peninsula Arts Gallery in Devon, UK, from 1 February to 29 March 2014. The timbre morphing system is used to transform live input captured at each sculpture with a discrete microphone array, by morphing towards noisy source signals that have been associated with each sculpture as part of a predetermined musical structure. The resulting morphed audio is then fed-back to the gallery via a five-channel speaker array. Visitors are encouraged to walk freely through the installation and interact with the sound world, creating unique audio morphs based on their own movements, voices, and incidental sounds.

This paper introduces the timbre synthesis technologies developed so far from the standpoint of composers. That is, the classification is based upon musical function. It also introduces major compositions in which new timbres synthesized by those technologies form the main structural core. Timbre synthesis technologies referred to here are granular synthesis, morphing and sound hybridization, which are based on various analysis/synthesis signal models such as LPC and sinusoidal models.