Neural network-based violinist's hand animation

Proceedings of the 5th international Workshop on Sensor-based Activity Recognition and Interaction

Motion Capture (MOCAP) Systems have been used to analyze body motion and postures in biomedicine, sports, rehabilitation, and music. With the aim to compare the precision of low-cost devices for motion tracking (e.g. Myo) with the precision of MOCAP systems in the context of music performance, we recorded MOCAP and Myo data of a top professional violinist executing four fundamental bowing techniques (i.e. Détaché, Martelé, Spiccato and Ricochet). Using the recorded data we applied machine learning techniques to train models to classify the four bowing techniques. Despite intrinsic differences between the MOCAP and low-cost data, the Myo-based classifier resulted in slightly higher accuracy than the MOCAPbased classifier. This result shows that it is possible to develop music-gesture learning applications based on low-cost technology which can be used in home environments for self-learning practitioners.

IEEE Transactions on Audio, Speech, and Language Processing, 2000

The objective of this research is to model the relationship between actions performed by a violinist and the sound which these actions produce. Violinist actions and audio are captured during real performances by means of a newly developed sensing system from which bowing and audio descriptors are computed. A database is built with this data and used to train a generative model based on neural networks. The model is driven by a continuous sequence of bowing and fingering controls and is able to generate their corresponding sequence of spectral envelopes. The model is used for synthesis, either alone as a purely spectral model, by filling the predicted envelopes with harmonic and noisy components, or coupled with a concatenative synthesizer, where the predicted envelopes are used as time-varying filters to transform the concatenated samples. The combination of sample concatenation with the timbre model allows for the preservation of sound quality inherent in samples, while providing a high level of control. Additionally, we perform an analysis of the violinist control space and the influence of the controls on the timbre.

Adaptive and Learning Systems, 1992

In this report we present our tools for prototyping adaptive user interfaces in the context of real-time musical instrument control. Characteristic of most human communication is the simultaneous use of classified events and estimated parameters. We have integrated a neural network object into the MAX language to explore adaptive user interfaces that considers these facets of human communication. By placing the neural processing in the context of a flexible real-time musical programming environment, we can rapidly prototype experiments on applications of adaptive interfaces and learning systems to musical problems. We have trained networks to recognize gestures from a Mathews radio baton, Nintendo Power Glove, and MDI keyboard gestural input devices. In one experiment, a network successfully extracted classification and attribute data from gestural contours transduced by a continuous space controller, suggesting their application in the interpretation of conducting gestures and musical instrument control. We discuss network architectures, low-level features extracted for the networks to operate on, training methods, and musical applications of adaptive techniques. 1. INTRODUCTION Human communication takes place through many different media such as audio, visual, or touch. In order for a someone to communicate an intention, they must channel it through the available media. Consider a conductor and an orchestra. During a musical performance, the only practical medium of communication is visual (a screaming conductor would detract from the performance). When a conductor wants the orchestra to play louder they might wave an open hand in the air and the orchestra plays louder according to the speed of his hand movement. Immediately following, the conductor makes a tight fist and moves it up and down and the orchestra changes the tempo of the music to match the speed of the conductor's moving fist. In this example, communication has taken place in the visual medium and intention is successfully translated in to and out of physical gestures. Characteristic of most human communication is the simultaneous use of classification information and parameter estimation. In the conductor/orchestra example, the orchestra members were able to classify the conductor's gestures and then extract information about them. Note that the classification needs to take place before the gesture can be fully interpreted. For example, the orchestra members cannot use the speed of the conductors moving hand information until they know what context it should be interpreted in. Key issues in human interfaces are determining how mappings between intention and gesture develop and determining what degree of adaptability to expect from each party. The problem can be simplified by first 244

2017

Violin performance is characterized by an intimate connection between the player and her instrument that allows her a continuous control of sound through a sophisticated bowing technique. A great importance in violin pedagogy is, then, given to techniques of the right hand, responsible of most of the sound produced. This study analyses the bowing trajectory in three different classical violin exercises from audio and motion capture recordings to classify, using machine learning techniques, the different kinds of bowing techniques used. Our results show that a clustering algorithm is able to appropriately group together the different shapes produced by the bow trajectories.

2017

From the first stage of learning new repertoire, string pedagogy aims to present violinists with effective strategies to achieve accurate intonation in performance. The search for new teaching and learning strategies, whilst running the risk of being seen as unconventional compared to more tried and tested methods (i.e. Suzuki), can on the other hand provide the tutor with new tools for enhancing their teaching practice. Western Australian violinist Fleur Challen has developed a method titled Finger Frames; a learning strategy that uses a colour coding system to prepare a violinist for changes in left hand position, reducing the cognitive load during the process of sight-reading new repertoire. In the field of psychology, research has been undertaken on artificial synaesthesia to indicate that when two particular senses are linked together, it eliminates a stage of processing between them, therefore reducing cognitive load. This research paper aims to document Challen’s Finger Frame...

Proceedings of the 2019 ACM Conference on Creativity & Cognition, 2019

This paper describes the evolution of a "hybrid violin" developed and played by the author and reflects on the process of developing its sensorimotor dynamics as a salient affordance of real-time digital signal processing. The hybrid violin consists of a technical ensemble of hardware and software elements, including an acoustic violin, microphone, custom sensor glove (alto.glove), and customized ergonomic shoulder rest embedded with voice coils for haptic feedback coupled to digital audio output. Through trial-and-error revision by the designer, sensory feedforward and feedback paths are effectively symmetrized, thereby engaging the claim of the "enactive" approach to cognition as a creative practice: perception drives action and action drives perception. The hybrid violin catalyzes improvised performance insofar as it is responsive to all gesture and embeds no assumptions about expressive intention. The system allows for experimentation with violin technique by continuously and intensively tracking gestural and auditory inputs.

2003

The human hand is a complex organ capable of both gross grasp and fine motor skills. Despite many successful high-level skeletal control techniques, animating realistic hand motion remains tedious and challenging. This paper presents research motivated by the complex finger positioning required to play musical instruments, such as the guitar. We first describe a data driven algorithm to add sympathetic finger motion to arbitrarily animated hands. We then present a procedural algorithm to generate the motion of the fretting hand playing a given musical passage on a guitar. The work here is aimed as a tool for music education and analysis. The contributions of this paper are a general architecture for the skeletal control of interdependent articulations performing multiple concurrent reaching tasks, and a procedural tool for musicians and animators that captures the motion complexity of guitar fingering.

Frontiers in Psychology

Gestures in music are of paramount importance partly because they are directly linked to musicians' sound and expressiveness. At the same time, current motion capture technologies are capable of detecting body motion/gestures details very accurately. We present a machine learning approach to automatic violin bow gesture classification based on Hierarchical Hidden Markov Models (HHMM) and motion data. We recorded motion and audio data corresponding to seven representative bow techniques (Détaché, Martelé, Spiccato, Ricochet, Sautillé, Staccato, and Bariolage) performed by a professional violin player. We used the commercial Myo device for recording inertial motion information from the right forearm and synchronized it with audio recordings. Data was uploaded into an online public repository. After extracting features from both the motion and audio data, we trained an HHMM to identify the different bowing techniques automatically. Our model can determine the studied bowing techniques with over 94% accuracy. The results make feasible the application of this work in a practical learning scenario, where violin students can benefit from the real-time feedback provided by the system.

2003

We introduce a software system for real-time classification of violin bow strokes (articulations). The system uses an electromagnetic motion tracking system to capture raw gesture data. The data is analyzed to extract stroke features. These features are provided to a decision tree for training and classification. Feedback from feature and classification data i s presented visually in an immersive graphic environment.

Proceedings of the 1st ACM SIGCHI International Workshop on Multimodal Interaction for Education

Though musicians tend to agree on the importance of practicing expressivity in performance, not many tools and techniques are available for the task. A machine learning model is proposed for predicting bowing velocity during performances of violin pieces. Our aim is to provide feedback to violin students in a technologyenhanced learning setting. Predictions are generated for musical phrases in a score by matching them to melodically and rhythmically similar phrases in performances by experts and adapting the bow velocity curve measured in the experts' performance. Results show that mean error in velocity predictions and bowing direction classification accuracy outperform our baseline when reference phrases similar to the predicted ones are available. CCS CONCEPTS • Applied computing → Sound and music computing; Interactive learning environments; • Computing methodologies → Instance-based learning;