DDX7: Differentiable FM Synthesis of Musical Instrument Sounds

Neural Computing and Applications

We present the Latent Timbre Synthesis (LTS), a new audio synthesis method using Deep Learning. The synthesis method allows composers and sound designers to interpolate and extrapolate between the timbre of multiple sounds using the latent space of audio frames. We provide the details of two Variational Autoencoder architectures for LTS, and compare their advantages and drawbacks. The implementation includes a fully working application with graphical user interface, called interpolate two, which enables practitioners to explore the timbre between two audio excerpts of their selection using interpolation and extrapolation in the latent space of audio frames. Our implementation is open-source, and we aim to improve the accessibility of this technology by providing a guide for users with any technical background.

IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2019

Sound synthesis is a complex field that requires domain expertise. Manual tuning of synthesizer parameters to match a specific sound can be an exhaustive task, even for experienced sound engineers. In this paper, we introduce InverSynth -an automatic method for synthesizer parameters tuning to match a given input sound. InverSynth is based on strided convolutional neural networks and is capable of inferring the synthesizer parameters configuration from the input spectrogram and even from the raw audio. The effectiveness InverSynth is demonstrated on a subtractive synthesizer with four frequency modulated oscillators, envelope generator and a gater effect. We present extensive quantitative and qualitative results that showcase the superiority InverSynth over several baselines. Furthermore, we show that the network depth is an important factor that contributes to the prediction accuracy.

2002

This dissertation introduces a connectionist model that maps perceptual controllers to synthesis parameters to allow for an intuitive and powerful musical control of audio synthesis. This model, or system, allows the extraction, abstraction, reproduction and transformation of relevant features of a musician's style. All the information is deduced exclusively from audio. No prior knowledge of the musician's performance is implied. We will first define the general underlying principles of the model. After, the neural network specific requirements for the learning and synthesis of sound information will be presented. With this foundation, the many advantages of using a principal component based synthesis will be argued. This dissertation concludes with an overview of some musical applications, including the re-tuning of a melody, pitch shifting, time stretching, crosssynthesis and compression.

2019

Sound synthesis is a complex field that requires domain expertise. Manual tuning of synthesizer parameters to match a specific sound can be an exhaustive task, even for experienced sound engineers. In this paper, we propose an automatic method for synthesizer parameters tuning to match a given input sound. The method is based on strided Convolutional Neural Networks and is capable of inferring the synthesizer parameters configuration from the input spectrogram and even from the raw audio. The effectiveness of our method is demonstrated on a subtractive synthesizer with four frequency modulated oscillators, envelope generator and a gater effect. We present extensive quantitative and qualitative results that showcase the superiority of our model over several baselines. Furthermore, we show that the network depth is an important factor that contributes to the prediction accuracy.

2000

A real-time synthesis engine that models and predicts the timbre of acoustic instruments based on perceptual features is presented. The timbre characteristics and the mapping between control and timbre parameters are inferred from recorded musical data. In the synthesis step, timbre data is predicted based on new control data enabling applications such as synthesis and cross-synthesis of acoustic instruments and

This paper presents a new system that allows for intuitive control of an additive sound synthesis model from perceptually relevant high-level sonic features. We suggest a general framework for the extraction, abstraction, reproduction and transformation of timbral characteristics of a sound analyzed from recordings. We propose a method to train, tune and evaluate our system in an automatic, consistent and reproducible fashion, and show that this system yields various original audio and musical applications.

Neural Computing and Applications, 2018

In addition to traditional tasks such as prediction, classification and translation, deep learning is receiving growing attention as an approach for music generation, as witnessed by recent research groups such as Magenta at Google and CTRL (Creator Technology Research Lab) at Spotify. The motivation is in using the capacity of deep learning architectures and training techniques to automatically learn musical styles from arbitrary musical corpora and then to generate samples from the estimated distribution. However, a direct application of deep learning to generate content rapidly reaches limits as the generated content tends to mimic the training set without exhibiting true creativity. Moreover, deep learning architectures do not offer direct ways for controlling generation (e.g., imposing some tonality or other arbitrary constraints). Furthermore, deep learning architectures alone are autistic automata which generate music autonomously without human user interaction, far from the objective of interactively assisting musicians to compose and refine music. Issues such as: control, structure, creativity and interactivity are the focus of our analysis. In this paper, we select some limitations of a direct application of deep learning to music generation, analyze why the issues are not fulfilled and how to address them by possible approaches. Various examples of recent systems are cited as examples of promising directions. * To appear in Special Issue on Deep learning for music and audio, Neural Computing & Applications, Springer Nature, 2018. 1 With many variants such as convolutional networks, recurrent networks, autoencoders, restricted Boltzmann machines, etc. [GBC16].

Proceedings of the 18th International Audio Mostly Conference

Tone Transfer is a novel deep-learning technique for interfacing a sound source with a synthesizer, transforming the timbre of audio excerpts while keeping their musical form content. Due to its good audio quality results and continuous controllability, it has been recently applied in several audio processing tools. Nevertheless, it still presents several shortcomings related to poor sound diversity, and limited transient and dynamic rendering, which we believe hinder its possibilities of articulation and phrasing in a real-time performance context. In this work, we present a discussion on current Tone Transfer architectures for the task of controlling synthetic audio with musical instruments and discuss their challenges in allowing expressive performances. Next, we introduce Envelope Learning, a novel method for designing Tone Transfer architectures that map musical events using a training objective at the synthesis parameter level. Our technique can render note beginnings and endings accurately and for a variety of sounds; these are essential steps for improving musical articulation, phrasing, and sound diversity with Tone Transfer. Finally, we implement a VST plugin for real-time live use and discuss possibilities for improvement. CCS CONCEPTS • Applied computing → Sound and music computing; • Computing methodologies → Neural networks; • Human-centered computing → Interaction techniques.

2005

A new approach to computer music instruments is described. Rather than sense control parameters from acoustic instruments (or non-acoustic instrument controllers), the sound of an acoustic instrument is used directly by a synthesis algorithm, usually replacing an oscillator. Parameters such as amplitude and pitch can control other aspects of the synthesis. This approach gives the player more control over details of the sound due to the use of the rich acoustic signal. Latency in sensing parameters, particularly pitch, is less of a problem because pitch information is carried directly by the acoustically generated signal. Several examples are described and the results of a subjective evaluation by musicians are presented. illustrates the configuration for what we call “Audio Signal Driven Sound Synthesis.” After describing related work, we describe two simple algorithms based on FM and subtractive synthesis. This leads to an oscillator-free (and pitch-estimation free) version of FM w...

Journal of The Audio Engineering Society, 2018

In this work, we propose a deep learning based method, namely, variational, convolutional recurrent autoencoders (VCRAE), for musical instrument synthesis. This method utilizes the higher level time-frequency representations extracted by the convolutional and recurrent layers to learn a Gaussian distribution in the training stage, which will be later used to infer unique samples through interpolation of multiple instruments in the usage stage. The reconstruction performance of VCRAE is evaluated by proxy through an instrument classifier, and provides significantly better accuracy than two other baseline autoencoder methods. The synthesized samples for the combinations of 15 different instruments are available on the companion website.