Gestural Interfaces

The role of human-computer interaction technologies has become a prominent factor that most can determine the successful introduction of new computer games. Players, in fact, wish to experience playful exchanges with the objects and characters that compose games. To reach this aim, new technologies have come into the play that comprise the use of video cameras and gesture recognition software. The great news is that such types of technologies could be exploited not only while playing at home on a console, but also in public spaces, thus broadening the use of games to new segments of customers. Nonetheless, to the best of our knowledge, neither relevant exemplars of such specific type of games (that can be played in public spaces) have yet emerged, nor extensive measurement studies exist regarding how players enjoy games in public immersive environments (e.g., fairs, museums and exhibits), the motivation being that those technologies that support completely hands-free gaming have been commercialized only very recently. Hence, our contribution with this article is twofold: on one side we want to illustrate the main design principles we have devised to design a gestural game to be played in a public space, based on novel hand following and gesture recognition techniques. On the other side we wish to report on real measurements we took when over one hundred players enjoyed our gestural game at the Shanghai 2010 World Expo.

Wearable technology is omnipresent to the user. Thus, it has the potential to be significantly disruptive to the user's daily life. Context awareness and intuitive device interfaces can help to minimize this disruption, but only when the sensing technology itself is not physically intrusive: i.e., when the interface preserves the user's homeostatic comfort. This work evaluates a novel foam- based

There are many ways in which digital games can present players with comic situations. They often contain jokes, funny dialogue, and ridiculous situations that provoke laughter. As an interactive experience, however, digital gameplay also presents possibilities for comedy that are not directly caused by the content presented in games, but are rather inherent to the way(s) players experience and relate to this content. In this chapter, I  discuss the comic potential that originates in the way players of digital games take on the dual position of being at once internal to the gameworld via fictional proxy and external to it as the actual person who is playing. I will describe how the comic attitude, as it has been discussed within the philosophy of humour, is intricately related to the dual position that players of digital games take on and how gameplay always entails the possibility of turning players into both a comic object and a laughing subject.

For published chapter, see https://link.springer.com/chapter/10.1007/978-3-030-88338-6_2

Gesture-based input devices for controlling music software should support expressivity. However, it is challenging to effectively design new gestural interactions that both exploit the capabilities of existing devices and afford user expressivity. To further the understanding of expressive interactions, we propose a demo that interactively supports users in the production of new gestural movements. Precisely, the demo implements an elicitation algorithm which rewards the originality of gesture dynamics to support users in the exploration of the space of motions they can perform. This demo is targeting 2D interactions supported by, for example, a mouse, trackpad or Sensel device. As users interact with their input device, movement dynamics (e.g. speed profile and orientation derivative) are computed and 2D strokes are segmented into small primitives. The originality of a new segment is then measured as the minimum distance to previously observed segments and fed back to the users as an audio cue; the greater the distance, the louder the feedback. Such sonification scheme encourages users to produce new behaviours. This elicitation produces a map of motion primitives wherein clusters identify prototypical user behaviour. The properties of these clusters, such as user specificity or time of invention, could be used to reflect on the user practice and identify new opportunities for interface design.

This paper explores how interactive media-specifically Kinesthesis 2.0, a system for real-time vocal modulation via gesture and facial expression-enhances vocal improvisation in drama performance when integrated with the AI improvisation system Somax2. Grounded in the embodied practices of Greek tragedy (schemata) and Kathakali theater (mudras and Navarasam), Kinesthesis 2.0 transforms performers into co-creators: closing the right eye triggers reverb, the left eye activates reverse delay, and opening the mouth scales multi-tap delay, while hand gestures map to pitch and harmonic shifts. Integrated with Somax2's generative clarinet responses, trained on Epirotic lament motifs, the system fosters human-AI co-improvisation, as demonstrated in Antigone's Lament. We describe the technical architecture (MediaPipe, Max/MSP), artistic applications, and broader implications for interactive storytelling, demonstrating how such tools redefine the performer's role in contemporary drama.

Numerous studies have tracked people's everyday use of digital devices, but without consideration of how such data might be of personal interest to the user. We have developed a personal tracking application that enables users to automatically monitor their 'screen time' on mobile devices (iOS and Android) and computers (Mac and Windows). The application interface enables users to combine screen time data from multiple devices. We trialled the application for 28+ days with 21 users, collecting log data and interviewing each user. We found that there is interest in personal tracking in this area, but that the study participants were less interested in quantifying their overall screen time than in gaining data about their use of specific devices and applications. We found that personal tracking of device use is desirable for goals including: increasing productivity, disciplining device use, and cutting down on use.

This paper presents D'Accord Guitar, an innovative environment for learning, editing and performing music. D'Accord Guitar can be seen as an Instrumental Performance System (IPS). In order to improve musical notation completeness, providing equally an intuitive user interface, this kind of system shows the music performance directly on a virtual instrument displayed on the computer screen. Unfortunately, the current IPS exhibit various limitations: they are hardly editable, or not editable at all, they are not enough interactive and their interface is not fully adequate for string instruments. D'Accord Guitar combines various features of the current IPS and other musical tools in order to provide a user-friendly multi-function interface.

One of the modern classical guitar’s biggest strengths is also one of its biggest weaknesses. The instrument depends on an equal temperament fretboard system that has lent itself to brilliant repertoire and ease of production, yet this same fretboard system does not lend itself to perfectly tuned notes and tuning systems not found in Eurocentric classical music. Many guitarists and luthiers have sought a fretboard system that marries the benefits of equal temperament with the rich possibilities of just intonation and other tuning systems. This paper presents two new fretboard designs and discuss their role in the history of guitar fretboard construction. Our first design is a fully functional fretboard made from the popular toy ‘Legos’ and solves many of the problems that previous designs had, including ease of use and production, affordability, and versatility. Our second design is a one-string prototype for an automatic (automated) fretboard that would allow performers to switch t...

Alternative control schemes for affecting the characteristics of audio signals have been designed and evaluated within the audio research community. The medium of virtual reality (VR) presents a unique method of sound source visualization using a headset which displays a virtual environment to the user, allowing users to directly control sound sources with minimal intermediary interference with a variety of different controllers. In order to provide insight into the design and evaluation of VR systems for audio mixing, the differences in subject preference between physical controllers and hand-and-gesture detection controls were investigated. A VR audio mixing interface was iteratively developed in order to facilitate a subject evaluation of some of the differences between these two control schemes. Ten subjects, recruited from a population of audio engineering technology undergraduate students, graduate students, and instructors, participated in a subjective audio mixing task. The results found that physical controllers outperformed the hand-and-gesture controls in each individual mean score of subject-perceived accuracy, efficiency, and satisfaction, with mixed statistical significance. No significant difference in task completion time for either control scheme was found. Additionally, the test participants largely preferred the physical controllers over the hand-and-gesture control scheme. There were no significant differences in the ability to make adjustments in general when comparing groups of more experienced and less experienced users. This study may provide useful contributing research to the wider field of audio engineering by providing insight into the design and evaluation of alternative audio mixing interfaces and further demonstrate the value of using VR to visualize and control sound sources in an articulated and convincing digital environment suitable for audio mixing tasks.

Parrots are exceptional among birds for their high levels of exploratory behaviour and manipulatory abilities. It has been argued that foraging method is the prime determinant of a bird's visual field configuration. However, here we argue that the topography of visual fields in parrots is related to their playful dexterity, unique anatomy and particularly the tactile information that is gained through their bill tip organ during object manipulation. We measured the visual fields of Senegal parrots Poicephalus senegalus using the ophthalmoscopic reflex technique and also report some preliminary observations on the bill tip organ in this species. We found that the visual fields of Senegal parrots are unlike those described hitherto in any other bird species, with both a relatively broad frontal binocular field and a near comprehensive field of view around the head. The behavioural implications are discussed and we consider how extractive foraging and object exploration, mediated i...

Examining nonverbal behavioral patterns specific to personality structures offers a valuable framework for understanding how psychopathological tendencies are manifested through body language. As a biopsychosocial being, the human individual is shaped not only by internal psychodynamic processes but also by the interaction of genetic predispositions (Plomin et al., 1994), early life experiences, traumatic events (van der Kolk, 2014), social context, and economic conditions (Engel, 1977). Therefore, in the evaluation of personality patterns, it is both scientifically and ethically essential to avoid reductionist, single-cause explanations and to adopt a multidimensional, dynamic, and contextual perspective (Millon & Davis, 1996; APA, 2013). Nonverbal communication-particularly body language-can occur beyond the scope of conscious control, thereby offering unique and direct insight into personality characteristics. In this context, the primary aim of this study is to systematically analyze the nonverbal indicators of different personality structures and to reveal how psychopathological patterns may become externally observable through bodily cues. The article discusses ten distinct personality types based on body language and behavioral indicators, addressing the potential forms of nonverbal expression characteristic to each. Additionally, the question of "How can these behavioral and nonverbal patterns be transformed into more objective data?" is positioned as a central focus of inquiry. In this regard, the study provides both interpretative insights based on qualitative observation and recommendations for future empirical research.

Mutualizing the body and the instrument offers a different way of thinking about designing embodied musical interfaces. This research presents the design of the Bod-yHarp, a wearable instrument that combines large bodybased gestures with the fine control of hand-based instruments. This reflects a desire to create a single interface that both captures expressive, dance-like, body movement as well as nuanced gestural interactions. The BodyHarp was not designed as a separate artifact, but rather it was crafted as an augmentation to the human body. This fusion seeks to enhance the sense of intimacy between the player and the instrument and carries a different type of aesthetic -like playing a traditional instrument (the harp) but as part of the body. In other words, the BodyHarp aims to capture creative body movement and placing it in an instrumental context. In doing so, we aim to blur the transition between two gestural mediums (dance and playing an instrument) by mutualizing them -or, in a sense, by designing the interface as a part of the body.

Nonverbal Communication: A Key Aspect of Human Interaction
Nonverbal communication is an essential component of meaning transmission between individuals, structured as a complex system of extralinguistic signals manifested through body kinetics, facial expressions, proxemics, eye contact, and paralinguistic elements. It plays a crucial role in modulating and interpreting verbal communication, as well as in constructing autonomous meanings. While it is a universal phenomenon, its expression and interpretation are culturally determined and subject to contextual variability. Numerous studies have demonstrated that nonverbal communication accounts for a significant portion of human interaction, contributing to the creation and maintenance of interpersonal relationships, conflict management, and the expression of individual and collective identity.
From a neurobiological perspective, nonverbal communication is deeply rooted in the evolutionary mechanisms of the human species, with its origins in primordial brain structures such as the limbic system and the amygdala, which regulate emotional responses to environmental stimuli. The ability to decode nonverbal signals is present from the first months of life and develops through interaction with the social environment, representing a fundamental element in the maturation of communicative and relational skills.
Furthermore, nonverbal communication plays a decisive role in regulating group dynamics, establishing leadership, and shaping perceptions of authority. Analyzing nonverbal cues allows for a better understanding of interlocutors’ intentions and the adaptation of one’s communication for greater effectiveness. Recent studies in social psychology have shown that a conscious use of nonverbal communication can significantly influence persuasion and conviction in negotiation and interpersonal contexts.
The Importance of Nonverbal Communication in Human Interactions
The study of nonverbal communication is critically important in the fields of cognitive sciences, social psychology, and applied linguistics. It plays a key role in regulating social interactions, providing implicit information that often surpasses the explanatory capacity of verbal language. Research in cognitive neuroscience and evolutionary psychology suggests that the human brain processes and responds to nonverbal signals more quickly and effectively than to linguistic ones, indicating that these signals constitute a primordial and instinctive mechanism of communication.
According to the studies of Albert Mehrabian, in emotionally charged interactions, verbal content has only a marginal impact compared to nonverbal elements, with body language and vocal tone playing a predominant role in conveying meaning. However, the influence of these variables differs based on situational and cultural factors, necessitating an interdisciplinary analysis to fully understand their communicative function.
From an applied perspective, nonverbal communication has significant implications in multiple professional and personal contexts. In interpersonal relationships, it enables the expression of empathy, affiliation, and social status, while in the workplace, it is a key element in negotiation, leadership, and personnel management. The ability to accurately interpret nonverbal signals can enhance the quality of interactions, facilitating cooperation and preventing misunderstandings. Moreover, the strategic use of nonverbal communication is crucial in media and advertising, where gestures, expressions, and postures are carefully studied to evoke emotional responses from the audience.
Nonverbal communication also plays a significant role in shaping individual and collective identity. Bodily signals, expressions, and nonverbal behaviors are used to assert a sense of belonging to a social group and to convey shared values and norms. Cultural differences in nonverbal communication are particularly relevant in the context of globalization and intercultural interactions, as they influence the perception and interpretation of communicative messages.
Objectives and Structure of the Essay
This essay aims to explore the various dimensions of nonverbal communication and its impact on relational dynamics, with a particular focus on similarities and differences between humans and non-human primates. The adopted approach will be multidisciplinary, combining insights from cognitive psychology, comparative ethology, semiotics, and social neuroscience to develop a comprehensive theoretical and applied framework.
The work will be structured into the following chapters:
Body Language and Gestures – An analysis of major gestural and postural codes, with particular attention to their pragmatic function and intercultural differences. Different types of gestures will be examined, categorized into emblems, illustrators, regulators, and adaptors, analyzing their role in conscious and unconscious communication.
Facial Expressions and Emotions – A study of microexpressions, their universality, and the role of primary emotions, referencing Paul Ekman’s research and neuroscientific studies on limbic circuits. The neurophysiological mechanisms of facial expressions and their impact on social perception will be discussed.
Eye Contact and Proxemics – An in-depth exploration of gaze management and interpersonal space, with a cross-cultural analysis of their variations. The function of eye contact in building trust and regulating conversation will be examined.
Voice Tone and Paralinguistic Elements – A study of the acoustic components of nonverbal communication, with a focus on intonation patterns, rhythm, and vocal modulation in conveying communicative intent. The implications of prosody in emotional perception and speaker credibility will also be explored.
Comparison Between Humans and Primates – A comparative analysis of human and primate communication methods, focusing on the neurobiological and evolutionary foundations of nonverbal communication. Ethological studies highlighting similarities between the gestural behavior of chimpanzees, bonobos, and humans will be discussed.
The objective of this essay is to provide an advanced analysis of nonverbal communication, outlining its central role in the construction of social meaning and communicative competence, with implications in clinical psychology, sociolinguistics, and cognitive neuroscience. Additionally, the essay will explore the practical applications of nonverbal communication in therapeutic, educational, and technological contexts, highlighting how emerging technologies such as artificial intelligence and facial recognition can contribute to the understanding and strategic use of nonverbal signals.

Vision Science Program School of Optometry Suppose the observer looks at and feels the surfaces simultaneously. The principle of maximum likelihood Berkeley, California 94720-2020 (ML) prescribes the strategy for combining visual and haptic estimates that produces the estimate of lowest variance [4-8]. If the visual and haptic estimates are Summary independent and normally distributed, that strategy is weighted summation Vision and haptics have different limitations and advantages because they obtain information by different S ˆVH ϭ w V S ˆV ϩ w H S ˆH, methods. If the brain combined information from the two senses optimally, it would rely more on the one w v ϭ 1/ v 2

Gesture authoring tools enable the rapid and experiential prototyping of gesture-based interfaces. We survey visual authoring tools for mid-air gestures and identify three paradigms used for representing and manipulating gesture information: graphs, visual markup languages and timelines. We examine the strengths and limitations of these approaches and we propose a novel paradigm to authoring location-based mid-air gestures based on space discretization.

With the growing usage of computer systems in daily life, a natural and intuitive Human Computer Interaction (HCI) method to support the embedding of computer systems in our environment seems necessary. Gestures are of utmost importance for the design of natural user interfaces. Hand gesture recognition to extract meaningful expressions from the human hand movements and postures is being used for different applications. However, the recognition of hand gestures that contain different hand poses can be challenging. In this ...

Elderly population is growing all over the globe. Novel human-computer interaction systems and techniques are required to fill the gap between elderly reduced physical and cognitive capabilities and the smooth usage of technological artefacts densely populating our environments. Gesture-based interfaces are potentially more natural, intuitive, and direct. In this paper, we propose a personalized hand pose and gesture recognition system (called HANDY) supporting personalized gestures and we report the results of two experiments with both younger and older participants. Our results show that by sufficiently training our system we can get similar accuracies for both younger and older users. This means that our gesture recognition system can accommodate the limitations of an ageing-hand even in presence of hand issues like arthritis or hand tremor.

Copyright & reuse City University London has developed City Research Online so that its users may access the research outputs of City University London's staff. Copyright © and Moral Rights for this paper are retained by the individual author(s) and / or other copyright holders. All material in City Research Online is checked for eligibility for copyright before being made available in the live archive. URLs from City Research Online may be freely distributed and linked to from other web pages. Versions of research The version in City Research Online may differ from the final published version. Users are advised to check the Permanent City Research Online URL above for the status of the paper. Enquiries If you have any enquiries about any aspect of City Research Online, or if you wish to make contact with the author(s) of this paper, please email the team at publications@city.ac.uk.Accessibility of 3D Game Environments for People with

This article presents new tools developed in the FAUST language to create musical interactions using electrophysiological signals as input. The tools developed are centered around signal processing and simulation of electrophysiological signals. These techniques are used to clean and process biosignals and subsequently provide real-time interactions to feed the control of sonic processes. The system provides modules that are musically expressive especially in the domain of spatial sound. These tools also allow to set up a testing environment by replacing the need of electrophysiological capture devices. The findings of this research provide a better understanding of how the FAUST language can be used in conjunction with physiological signals and brings to light interesting opportunities to explore further possibilities in music creation in an open source environment with the possibility of multitarget compilation, allowing our modules to be used either in such softwares as Max[10] or embedded in microcontrollers.

Bien que dramatique, la perte de données contextuelles afférentes à une partie du matériel archéologique acquis par les Musées royaux d’Art et d’Histoire de Bruxelles ne constitue pas nécessairement une fatalité. À titre d’exemple, bon nombre d’artéfacts sonores précolombiens conservés au Musée Art & Histoire (MAH) et au Musée des Instruments de Musique (MIM) présentent un état de conservation exceptionnel. Au-delà des éléments iconographiques, la préservation des détails de facture et des propriétés acoustiques originales de ces instruments ouvre de nouvelles perspectives méthodologiques à l’archéomusicologue. Le présent article constitue un bref plaidoyer en faveur d’une réappropriation physique des artéfacts sonores anciens, ceci afin de mettre en lumière certains aspects éphémères de la culture matérielle trop souvent négligés par les archéologues. Parmi ceux-ci, le matériau acoustique en tant que tel, la posture et les gestes performatifs de l’interprète ainsi que la configuration organologique des instruments sont examinés dans leurs rapports mutuels à travers l’analyse d’une sélection de neuf aérophones provenant des territoires actuellement couverts par la République du Costa Rica et la province panaméenne de Chiriquí. Dans ce cadre, nous proposons une série d’hypothèses relatives à la forme originelle de jouer ces instruments, à la fonction d’un type d’embouchure spécifiquement costaricien, ainsi qu’aux notions d’affordances et d’interfaces gestuelles.

Understanding the mechanics of upper extremity motions is essential for improving the quality of life for individuals with neuro-musculoskeletal dysfunctions. This study compares muscle force predictions during upper extremity motions using two widely utilized computational tools: AnyBody Modeling System™ (AMS) and OpenSim. Both software programs were used to simulate static elbow flexion, shoulder flexion, and shoulder abduction maneuvers based on a generic model of the 50th percentile adult male.

Results indicated that while both software predicted similar roles for major muscles such as the brachialis during elbow flexion, significant discrepancies were observed in muscle force predictions for the triceps, deltoids, rotator cuff muscles, serratus, and trapezius. Notably, OpenSim predicted high forces in the triceps during shoulder motions, inconsistent with literature, likely due to the absence of trapezius and serratus muscles in its model.

The study underscores the importance of selecting appropriate models and parameters for accurate biomechanical simulations. Overall, AnyBody was found to be more viable for predicting muscle forces in upper extremity motions due to its comprehensive muscle modeling capabilities.

Gesture recognition is a field of Computer Science which deals with interpreting the human gestures via running some mathematical algorithms on a computing device. Gestures can be any human motion that can be generated either by hands, face, eyes, etc. Hand gesture recognition system propagates the elevated interaction experience for the users as it integrates the virtual and real world object. The traditional devices that are used for interaction purpose are often controlled in an orthodox manner, which at times become quite cumbersome. By using gesture recognition, the focus will be on replacing the age-old hardware which we are currently using, with some more natural, lucid, human-centered means of interacting. The color detection based interaction interface being scripted here can be appreciably applied towards many applications like Human-Computer interaction (HCI), games, virtual Reality. Though the present paper's prime focus is on the GAMES. With the advent of gaming in ...

With the evolution of computers, there was also an evolution in the universe of users of such equipment. The audience became heterogeneous, comprising people of different skill levels. This fact has generated the need to facilitate interaction between computer and users, called usability. Usability testing should be conducted throughout the development of the system because they help identify the needs of users and minimize the stress of the system acceptance. The paper presents a usability method developed specifically for the Brazilian Institute of Geography and Statistics (IBGE), called Scenario and Task Based Interviews (STBI). This method consists of a flexible form of usability testing and it has been adapted to be applied in the context of the IBGE, with the participation of institute interviewers, using PDAs (Personal Digital Assistant) for their data collection. The proposed method consists of two steps: first, the user is observed using the PDA during the field trip and then the user participates in an usability interview with the use of a portable laboratory. Participants are presented to several scenarios, which are explanatory texts that simulate common situations that occur in field work and tasks for him to use the application on the PDA. The entire interaction is videotaped. The analysis of recorded images allowed the researchers to detect several usability problems, to varying degrees.

This paper presents a robust and real-time hand posture recognition system. To obtain this, key elements of the proposed system contain an user-guide scheme and a kernel-based hand posture representation. We firstly describe a three-stage scheme to train an end-user. This scheme aims to adapt environmental conditions (e.g., background images, distance from device to hand/human body) as well as to learn appearance-based features such as hand-skin color. Thanks to the proposed user-guide scheme, we could precisely estimate heuristic parameters which play an important role for detecting and segmenting hand regions. Based on the segmented hand regions, we utilize a kernel-based hand representation in which three levels of feature are extracted. Whereas pixel-level and patch-level are conventional extractions, we construct image-level which presents a hand pyramid structure. These representations contribute to a Multi-class support vector machine classifier. We evaluate the proposed system in term of the learning time versus the robustness and real time performances. Averagely, the proposed system requires 14 s in advanced to guide an end-user. However, the hand posture recognition rate obtains 91.2% accuracy. Performance of the proposed system is comparable with state-of-the-art methods (e.g. Pisharady et al., 2012) but it is a real time system. To recognize a posture, its computational cost is only 0.15 s. This is significantly faster than works in Pisharady et al. (2012), which required approximately 2 min. The proposed methods therefore are feasible to embed into smart devices, particularly, consumer electronics in domain of home-automation such as televisions, game consoles, or lighting systems.

Metodologias de concepção de interface do usuário apresentam em seu fluxo atividades associadas ao uso de protótipos de interface. O uso de técnicas de prototipagem traz vantagens tanto para o processo quanto para o produto final obtido. Os protótipos de interface podem ser classificados em níveis de fidelidade ao produto final: baixa, média e alta fidelidade. Este artigo descreve SMILE, uma ferramenta para geração automática, edição e simulação de protótipos de média-fidelidade representados na forma de esboços de interface.

We present an overview of machine learning (ML) techniques and their application in interactive music and new digital instrument design. We first provide the non-specialist reader an introduction to two ML tasks, classification and regression, that are particularly relevant for gestural interaction. We then present a review of the literature in current NIME research that uses ML in musical gesture analysis and gestural sound control. We describe the ways in which machine learning is useful for creating expressive musical interaction, and in turn why live music performance presents a pertinent and challenging use case for machine learning.

When performing a piece, a pianist's interpretation is communicated both through the sound produced and through body gestures. We present PiaF (Piano Follower), a prototype for augmenting piano performance by measuring gesture variations. We survey other augmented piano projects, several of which focus on gestural recognition, and present our prototype which uses machine learning techniques for gesture classification and estimation of gesture variations in real-time. Our implementation uses the Kinect depth sensor to track body motion in space, which is used as input data. During an initial learning phase, the system is taught a set of reference gestures, or templates. During performance, the live gesture is classified in real-time, and variations with respect to the recognized template are computed. These values can then be mapped to audio processing parameters, to control digital effects which are applied to the acoustic output of the piano in real-time. We discuss initial tests using PiaF with a pianist, as well as potential applications beyond live performance, including pedagogy and embodiment of recorded performance.

In this paper we present two datasets of instrumental gestures performed with expressive variations: ve violinists performing standard pedagogical phrases with variation in dynamics and tempo; and two pianists performing a repertoire piece with variations in tempo, dynamics and articulation. We show the utility of these datasets by highlighting the di erent movement qualities embedded in both datasets. In addition, for the violin dataset, we report on gesture recognition tests using two state-of-the-art realtime gesture recognizers. We believe that these resources create opportunities for further research on the understanding of complex human movements through computational methods.

When performing a piece, a pianist's interpretation is communicated both through the sound produced and through body gestures. We present PiaF (Piano Follower), a prototype for augmenting piano performance by measuring gesture variations. We survey other augmented piano projects, several of which focus on gestural recognition, and present our prototype which uses machine learning techniques for gesture classification and estimation of gesture variations in real-time. Our implementation uses the Kinect depth sensor to track body motion in space, which is used as input data. During an initial learning phase, the system is taught a set of reference gestures, or templates. During performance, the live gesture is classified in real-time, and variations with respect to the recognized template are computed. These values can then be mapped to audio processing parameters, to control digital effects which are applied to the acoustic output of the piano in real-time. We discuss initial tes...

Elderly population is growing all over the globe. Novel human-computer interaction systems and techniques are required to fill the gap between elderly reduced physical and cognitive capabilities and the smooth usage of technological artefacts densely populating our environments. Gesture-based interfaces are potentially more natural, intuitive, and direct. In this paper, we propose a personalized hand pose and gesture recognition system (called HANDY) supporting personalized gestures and we report the results of two experiments with both younger and older participants. Our results show that by sufficiently training our system we can get similar accuracies for both younger and older users. This means that our gesture recognition system can accommodate the limitations of an ageing-hand even in presence of hand issues like arthritis or hand tremor.

Not just the screens of television (cathode, plasma, projection, LCD, PLED); not just the flow and the style of its programming; not just its choices of network and channel; not just broadcast, cable, satellite, or internet-streaming video imaging; not just live or time-shifted; not just console games' promises of interaction and immersion; not just the material presence of the screenbut the metaphysical largesse of a remote visuality. Televisual space is both the space of the televisual and the changes produced by the televisual to space itself.

User interfaces that utilise human gestures as input are becoming increasingly prevalent in diverse computing applications. However, few designers possess the deep insight, awareness and experience regarding the nature and usage of gestures in user interfaces to the extent that they are able to exploit the technological affordances and innovate over them. We argue that design students, who will be expected to envision and create such interactions in the future, are constrained as such by their habits that pertain to conventional user interfaces. Design students should gain an understanding of the nature of human gestures and how to use them to add value to UI designs. To this end, we formulated an 'awareness course' for design students based on concepts derived from mime art and creative drama. We developed the course iteratively through the involvement of three groups of students. The final version of the course was evaluated by incorporating the perspectives of design educators, an industry expert and the students. We present the details of the course, describe the development process, and discuss the insights revealed by the evaluations.

In this paper, we introduce an evaluation of accelerometer-based gesture recognition algorithms in user dependent and independent cases. Gesture recognition has many algorithms and this evaluation includes Hidden Markov Models, Support Vector Machine, K-nearest neighbor, Artificial Neural Network and Dynamic Time Warping. Recognition results are based on acceleration data collected from 12 users. We evaluated the algorithms based on the recognition accuracy related to different number of gestures from two datasets. Evaluation results show that the best accuracy for 8 and 18 gestures is achieved with dynamic time warping and K-nearest neighbor algorithms.

Expressiveness, which we define as the extent to which rich and complex intent can be conveyed through action, is a vital aspect of many human interactions. For instance, paint on canvas is said to be an expressive medium, because it affords the artist the ability to convey multifaceted emotional intent through intricate manipulations of a brush. To date, touch devices have failed to offer users a level of expressiveness in their interactions that rivals that experienced by the painter and those completing other skilled physical tasks. We investigate how data about hand movementprovided by a motion sensor, similar to those found in many smart watches or fitness trackers-can be used to expand the expressiveness of touch interactions. We begin by introducing a conceptual model that formalizes a design space of possible expressive touch interactions. We then describe and evaluate Expressy, an approach that uses a wrist-worn inertial measurement unit to detect and classify qualities of touch interaction that extend beyond those offered by today's typical sensing hardware. We conclude by describing a number of sample applications, which demonstrate the enhanced expressive interaction capabilities made possible by Expressy.

La unión entre arte y ciencia ha sido una constante desde los inicios de la creación plástica. Ha funcionado como el motor que ha permitido el desarrollo de nuevas vías de expresión. Utilizando como base ese vínculo el presente artículo trata de establecer un estudio de la práctica de la música visual acotada a la imagen proyectada y su vinculación con la música en directo. Mediante un recorrido histórico se recuperan aquellos hitos que han configurado estas proyecciones y que han construido lo que hoy se entiende como disc joquey. Se partirá de la pintura y el cine como elementos referenciales para explicar la aparición y desarrollo de los dispositivos conocidos como órgano color y sus posteriores evoluciones. La investigación toma como referentes aquellos avances técnicos que han permitido a los artistas conseguir nuevos logros visuales, pero también se atenderá el caso contrario, la necesidad plástica como elemento motivacional de hallazgos técnicos.
Finalmente se propone un estudio del VJing como práctica de recuperación de ese binomio entre arte y ciencia y la evolución lógica de las primeras concepciones de música visual asociadas al momento actual.

The use of motion controllers and cameras such as Microsoft Kinect as a gaming peripheral have increased the immersion of computer games today. Moving to using gestures instead of conventional handheld game controllers has increased the interest of players. Xbox 360, Xbox-One and the games that have been developed to be used with Kinect enable users to play using gestures, but this limits us to play our old favorite games and the new non-Kinect based games with same conventional peripherals. To overcome this limitation, we have developed a user friendly interface in MATLAB that replaces keyboard and mouse controls with our own designed gestures and enables us to play all non-Kinect based games using Kinect. For testing and demonstration purpose, we have applied the developed interface to three games which are Counter-Strike Global Offense, Candy Crush and Asphalt 8 Airborne.

There is a growing interest in `unlocking' the motor skills of expert musicians. Motivated by this need, the main objective of this paper is to present a new way of modeling expressive gesture variations in musical performance. For this purpose, the 3D gesture recognition engine `x2Gesture' (eXpert eXpressive Gesture) has been developed, inspired by the Gesture Variation Follower, which is initially designed and developed at IRCAM in Paris and then extended at Goldsmiths College in London. x2Gesture supports both learning of musical gestures and live performing, through gesture sonification, as a unified user experience. The deeper understanding of the expressive gestural variations permits to define the confidence bounds of the expert's gestures, which are used during the decoding phase of the recognition. The first experiments show promising results in terms of recognition accuracy and temporal alignment between template and performed gesture, which leads to a better f...

A large variety of musical instruments, either acoustical or digital, are based on a keyboard scheme. Keyboard instruments can produce sounds through acoustic means but they are increasingly used to control digital sound synthesis processes with nowadays music. Interestingly, with all the different possibilities of sonic outcomes, the input remains a musical gesture. In this paper we present the conceptualization of a Natural User Interface (NUI), named the Intangible Musical Instrument (IMI), aiming to support both learning of expert musical gestures and performing music as a unified user experience. The IMI is designed to recognize metaphors of pianistic gestures, focusing on subtle uses of fingers and upper-body. Based on a typology of musical gestures, a gesture vocabulary has been created, hierarchized from basic to complex. These piano-like gestures are finally recognized and transformed into sounds.

Resumo: O cidadão sénior pode beneficiar da utilização de Tecnologias de Informação e Comunicação (TIC) apesar da existência de diversas barreiras e dificuldades que sente no seu uso. O Skype é uma ferramenta de comunicação síncrona e assíncrona que permite realizar chamadas e trocar mensagens. Os objetivos deste estudo exploratório são: i) avaliar a usabilidade do Skype; ii) verificar e descrever as dificuldades que os seniores detetaram durante a sua utilização; e iii) oferecer sugestões para a melhoria de sua interface. Trata-se de um estudo exploratório com recurso à observação participante e guião de tarefas. As tarefas foram avaliadas consoante os erros que ocorriam, a necessidade de ajuda e cumprimento das tarefas que posteriormente foram comparadas a heurísticas específicas. Os participantes apresentaram dificuldades referentes à aprendizagem, eficiência e na resolução dos erros. De um modo geral, os seniores não conseguiram finalizar as tarefas propostas, o que sugere que o Skype necessita de algumas alterações para tornar-se realmente fácil de utilizar e ao alcance de todos.

Game Books can oer a well-written, but non-linear story, as readers always have to decide, how to continue after reading a text passage. It seems very logical to adopt such a book to investigate interaction paradigms for an interactive storytelling scenario. Nevertheless, it is not easy to keep the player motivated during a long-winded narrated story until the next point of intervention is reached. In this paper we tested dierent methods of implementing the decision process in such a scenario using speech input and tested it with 26 participants during a two player scenario. This revealed that with an omitted on-screen prompt the application was less easy to use, but caused considerably more user interaction. We further added additional interactivity with so-called Quick Time Events (QTEs). In these events, the player has a limited amount of time to perform a specic action after a corresponding prompt appeares on screen. Dierent versions of QTEs were implemented using Full Body Tracking with Microsoft Kinect, and were tested with another 18 participants during a two player scenario. We found that Full Body Gestures were easier to perform and, in general, preferred to controlling a cursor with one hand and hitting buttons with it.

For improving full body interaction in an interactive storytelling scenario, we conducted a study to get a user-defined gesture set. 22 users performed 251 gestures while running through the story script with real interaction disabled, but with hints of what set of actions was currently requested by the application. We describe our interaction design process, starting with the conduction of the study, continuing with the analysis of the recorded data including the creation of gesture taxonomy and the selection of gesture candidates, and ending with the integration of the gestures in our application.

There is a growing interest in 'unlocking' the motor skills of expert musicians. Motivated by this need, the main objective of this paper is to present a new way of modeling expressive gesture variations in musical performance. For this purpose, the 3D gesture recognition engine 'x2Gesture' (eXpert eXpressive Gesture) has been developed, inspired by the Gesture Variation Follower, which is initially designed and developed at IRCAM in Paris and then extended at Goldsmiths College in London. x2Gesture supports both learning of musical gestures and live performing, through gesture sonification, as a unified user experience. The deeper understanding of the expressive gestural variations permits to define the confidence bounds of the expert's gestures, which are used during the decoding phase of the recognition. The first experiments show promising results in terms of recognition accuracy and temporal alignment between template and performed gesture, which leads to a better fluidity and immediacy and thus gesture sonification.

There is a growing interest in `unlocking' the motor skills of expert musicians. Motivated by this need, the main objective of this paper is to present a new way of modeling expressive gesture variations in musical performance. For this purpose, the 3D gesture recognition engine `x2Gesture' (eXpert eXpressive Gesture) has been developed, inspired by the Gesture Variation Follower, which is initially designed and developed at IRCAM in Paris and then extended at Goldsmiths College in London. x2Gesture supports both learning of musical gestures and live performing, through gesture sonification, as a unified user experience. The deeper understanding of the expressive gestural variations permits to define the confidence bounds of the expert's gestures, which are used during the decoding phase of the recognition. The first experiments show promising results in terms of recognition accuracy and temporal alignment between template and performed gesture, which leads to a better f...

A large variety of musical instruments, either acoustical or digital, are based on a keyboard scheme. Keyboard instruments can produce sounds through acoustic means but they are increasingly used to control digital sound synthesis processes with nowadays music. Interestingly, with all the different possibilities of sonic outcomes, the input remains a musical gesture. In this paper we present the conceptualization of a Natural User Interface (NUI), named the Intangible Musical Instrument (IMI), aiming to support both learning of expert musical gestures and performing music as a unified user experience. The IMI is designed to recognize metaphors of pianistic gestures, focusing on subtle uses of fingers and upper-body. Based on a typology of musical gestures, a gesture vocabulary has been created, hierarchized from basic to complex. These piano-like gestures are finally recognized and transformed into sounds.

Su trabajo de investigación aborda el arte, el diseño, la política y la tecnología, teniendo como focos de interés Internet y su visualidad, el Tactical Media y el diseño interactivo. Es autora de "Aprender y disentir, la experiencia en Summer School de la Manifesta 11 y la ZHdK en Zúrich, Suiza" (Artediseño, n° 3) y "El espacio-frontera y su inscripción en el cuerpo migrante. La operación contra-performativa de la obra Transborder Immigrant Tool" (Arte y Políticas de Identidad, vol. 18

In this paper, the authors describe working with and on the T-Tree, a device that integrates multiple instances of a gestural controller known as the T-Stick. The T-Tree is used in two public performance contexts; the results of those performances are summarized, potential improvements to the design of the hardware and software are introduced, and issues are identified. Improvements in the T-Tree from the first version are also discussed. Finally, the authors present future design improvements for the T-Tree 2.0.

With emerging opportunities for using Brain-Computer Interaction (BCI) in gaming applications, there is a need to understand the opportunities and constraints of this interaction paradigm. To complement existing laboratorybased studies, there is also a call for the study of BCI in real world contexts. We present such a real world study of a simple BCI game called MindFlex ® , played as a social activity in the home. In particular, drawing on the philosophical traditions of embodied interaction, we highlight the importance of considering the body in BCI and not simply what is going on in the head. The study shows how people use bodily actions to facilitate control of brain activity but also to make their actions and intentions visible to, and interpretable by, others playing and watching the game. It is the public availability of these bodily actions during BCI that allows action to be socially organised, understood and coordinated with others and through which social relationships can be played out. We discuss the implications of this perspective and findings for BCI.