CG animation for piano performance

2014

Movements of the body and especially of the hand are extremely complex, and are driven by several processes of biomechanical and neuroplastic types, among others. Their study presents a great interest in Human-Computer Interaction (HCI), in sportive and musical education, as well as in medicine and reeducation. The aim of this research was to capture and analyze highly complex gestures performed by experts in hand motor control, such as musicians. Musical gestures are highly formalized by indications generally noted on a musical score, allowing a more focused and thus objective study of the hands movements. Piano playing, which requires rather precise and coordinated movements, was therefore chosen as our case study. In this research, we developed an optical motion capture setup that allowed the capture of pianists' expert gestures in playing complex piano pieces. The setup consisted of 12 infrared cameras aimed at the keyboard and capturing in 3D all Degrees Of Freedom (DOF) of both hands, by means of 26 retro-reflective markers placed on each hand. With this setup, we captured performances by four pianists with different expertise levels, playing six different piano pieces and an exercise. An example of the successful capture of the Fantasy in C Major by Robert Schumann can be found here: http://youtu.be/lNwWHlkjNJ4. The exercise allowed us to analyze redundant DOF solicitations in a basic keystroke gesture according to different tempi and movement instructions. This analysis showed that pianists have their own movement signature in natural playing. It also showed that the same pianist can use its redundant DOF of the arm and hand differently, and thus play the same musical pattern in countless different ways. We also carried out a Principal Component Analysis (PCA) of the hands gestures, based on the six pieces performed by the four pianists. Results of PCA showed the power of this tool for dimensionality reduction, and representation of pianists' expert gestures. Eight Principal Components (PC) were generally sufficient to represent 95% of the 23 DOF of the hand (and only three PC for 80%). Furthermore, PCA allowed us to decompose hands movements into eigengestures associated with each PC (see http://youtu.be/6qIHn9UUJW4). Finally, both analyses of the exercise and of the piano pieces seemed to show that a more trained pianist has a finer hand motor control, and in particular, a finer balance between the two hands. Results of this study show that PCA is a powerful tool for motion analysis and could be applied in different fields such as HCI, and sportive and musical education. Furthermore, in a clinical context, PCA could provide an assessment tool for patients suffering from motor disability.

Fingering is an important aspect of piano performance be-cause it affects a pianist's musicality. If audience members, especially pianists, shared the real-time fingering of the per-former in a concert, they would feel a sense of togetherness, and this could help them in learning techniques of profes-sional piano performance. To realize such a possibility, the goal of our study is to construct a real-time fingering de-tection system for piano performance. Our system achieves real-time fingering detection by integrating a simple camera-based image-detection method and musical rules. We have developed a prototype system and evaluated in effectiveness by actual use.

2003

mv, a l~n i a~l r a l i . c s h a q (~~~~~~~i~~.~~~~~. m v , norzaiha.norhan(i7!inn.etln.nlv Abstract-The history of animation is generally referred to early production of traditional animation. It has begun in the early 1900, "Animalion is the essence of animation " as explained by Halas [I]. It is a series of images that appear to be in motion. It can be either traditional animation or computer animation. In addition Greenberg [23, explains "Animation is a deliberately interpreted "illusion of l$e. " I f 110s been practiced for the last 7 0~e a r s in both drawn and stop motion animnotion, and as such, is not an attempt to mimic kurnan and animal l$e exactly." Traditionally animation is produced by drawing illusion of movement created by photographing a series of individual drawings on successive frames on film. Gleicher [3], explains "Animation is a uniquely expressive art form: it provides the creator with conrrol over both the appmrance and the rirovcnienl of clzarricfers and objects. This gives artists tremendous freedom. which when well used, can create works with iremendous impact."

2016

2000

We present a system for the animation of human hand that plays violin. Neural network controls the hand movement. We make use of an optimization method to generate the examples for the neural network training. The musical decision of which finger to use is automatically made by best first search. We will show that the movements of violinist's hands are physically and musically feasible, and that the musical decisions are consistent with those recommended in the violin pedagogy. A description of system, the results of the decisions, and the animations are presented.

The grand piano action is modeled as a set of four rigid bodies using Kane's method. Computerized symbol manipulation is utilized to streamline the formulation of the equations of motion so that several models can be considered, each of increasing detail. Various methods for checking the dynamical model thus derived are explored. A computer animation driven by simulation of the equations of motion is compared to a high-speed video recording of the piano action moving under a known force at the key. For quantitative evaluation, the velocities and angular velocities of each of the bodies are extracted from the video recording by means of digitization techniques. The aspects of the model of particular interest for emulation by a controlled system, namely, the mechanical impedance at the key and the velocity with which the hammer strikes the string, can be studied in the equations of motion and compared to empirical data.

International Journal of Hybrid Information Technology, 2015

As mobiles become cheaper and gain popularity, mobile applications are developing quickly and trying to utilize more techniques to make these applications more efficient and usable. Augmented reality is a live direct or indirect view of a physical, real-world environment whose elements are augmented (or supplemented) by computer-generated sensory input such as sound, video, graphics or GPS data. This research develops an android mobile application that utilizes real-time image processing techniques to provide simulated functionality to a hand drawing of piano keys. A piano keyboard with fourteen white keys and ten black keys is drawn used a black thick pen on paper, and the mobile is set in an angel that gives the optimum view of the piano keyboard when the camera is turned on. When the application is started, a camera shot of the piano, without the presence of hands or fingers in the view, is taken. The user can then start playing on his drawn piano. The application detects keystrokes and decides which key is pressed; and then plays the corresponding tone. The application is developed using OpenCV library which focuses on real-time image processing and computer vision, since the target devices with android platform, a Java program is used to port the C++ code to android. The application is tested on multiple android-based devices with different specifications under good lighting conditions. The empirical results are impressive and comparable across different devices, despite all changes in lighting and background; all devices exhibited the same level of accuracy in detecting fingers and drawn pianos.

Animation, which is basically a form of pictorial presentation, has become the most prominent feature of technology-based learning environments. It refers to simulated motion pictures showing movement of drawn objects. Recently, educational computer animation has turned out to be one of the most elegant tools for presenting multimedia materials for learners, and its significance in helping to understand and remember information has greatly increased since the advent of powerful graphics-oriented computers. In this book chapter we introduce and discuss the history of computer animation, its well-known fundamental principles and some educational applications. It is however still debatable if truly educational computer animations help in learning, as the research on whether animation aids learners' understanding of dynamic phenomena has come up with positive, negative and neutral results. We have tried to provide as much detailed information on computer animation as we could, and w...

2015

This paper presents a study into music analysis, motion analysis and the integration of music and motion to form natural human motion in a virtual environment. Motion capture data is extracted to generate a motion library; this places the digital motion model at a fixed posture. The first step in this process is to configure the motion path curve for the database and calculate the possibility that two motions were sequential through the use of a computational algorithm. Every motion is then analyzed for the next possible smooth movement to connect to, and at the same time, an interpolation method is used to create the transitions between motions to enable the digital motion models to move fluently. Lastly, a searching algorithm sifts for possible successive motions from the motion path curve according to the music tempo. It was concluded that the higher ratio of rescaling a transition, the lower the degree of natural motion.

Siggraph 2009 Talks, 2009