In rich pervasive environments, there will be numerous opportunities for end users to dynamically... more In rich pervasive environments, there will be numerous opportunities for end users to dynamically create services of interest by causing two or more devices or resources to interoperate together, often under changing circumstances. In general, users find this kind of process hard to manage. Existing programming architectures make the situation difficult to address in a principled, scaleable way. Users find it hard to tackle such problems via devices with small, resource-poor user interfaces. It is proposed that a good theoretical basis for addressing an essential aspect of all of these problems is the theory of Direct Combination. When the Direct Combination framework, based on the theory, is applied to spontaneous interactions, the user interface can be made relatively simple, and the amount of search required by the user to specify desired actions can be greatly reduced. We present Direct Combination (DC) and the new interaction techniques it gives rise to for pervasive environments. We consider two different support architectures. We argue that one of these, the role-based architecture, has particularly good properties for modelling rapidly changing pervasive environments, and for highly distributed implementations. We demonstrate how the concept of viewpoints can be used to focus, filter and afford operations, and how this can be well supported by the role-based architecture.
This paper forms part of a preliminary survey for work on the application of artificial intellige... more This paper forms part of a preliminary survey for work on the application of artificial intelligence theories and techniques to the learning of music composition skills. The paper deals with present day applications of computers to the teaching of music and speculations about how artificial intelligence might be used to foster music composition in the future. The field of music composition makes use of computer aided instruction in the areas of music theory, music history, and aural testing. Music Logo, the use of the programming language, Logo, to create music, is another application of computer technology to music. Computerized musical instruments allow students to compose and play music at their own level; to listen and adapt these compositions; and to analyze existing music for pitch, note intervals and values, and pattern recognition. Interactive videogames provide untrained students with opportunities to compose music and hear it immediately and to teach musical transformations through simple visual manipulation of music. In the future, artificial intelligence may make modest contributions to support the learning of music composition in the areas of intelligent tutors that focus on music theory, aural training, harmonization, and some highly formalized and artificial styles of composition. Intelligent tools in the form of editors, instruments, and analytical aids along with educational games also could help students learn about music composition. The document contains 16 figures. (DB)
International Computer Music Conference, Sep 11, 2014
In popular music genres, typical songs are pre-composed and leave little or no space for improvis... more In popular music genres, typical songs are pre-composed and leave little or no space for improvisation during a live performance. That applies for the performing musicians as well as for the spectators in terms of interactive audience participation. In this study we question these improvisational limits and try to identify strategies for involving the audience as an additional and unpredictable factor in a pre-composed rock song. To do so we composed "Experimence" guided by the standard practice of song writing. The song was premiered at a public live concert where the audience could collaboratively participate in real-time by playing with a balloon together throughout the song. Using a wizard of oz technique, the movements of the balloon influenced the live music played by the pianist. We reflect across this experience and present notable issues raised during the composition, rehearsals and the actual performance. We then classify these aspects as abstract variables of consideration for a composition meant to promote such audience participation. We propose this proof of concept as a starting point for further discussion, suggesting that a song such as Experimence can be a unique and individual piece of music every time it is played although largely pre-composed.
International journal of human-computer studies, Aug 1, 2019
Nonlinear dynamical processes play a central role in many acoustic instruments, yet they rarely f... more Nonlinear dynamical processes play a central role in many acoustic instruments, yet they rarely feature in digital instruments, and are little understood from an interaction design perspective. Such processes exhibit behaviours that are complex, timedependent, and chaotic, yet in the context of acoustic instruments can facilitate interactions that are explorable, learnable and repeatable. This suggests that these processes merit deeper investigation for digital music interaction design. Two studies are presented which investigate user interaction with nonlinear dynamical musical tools. A lab-based study used four purpose-built digital musical instruments to test interaction designs featuring nonlinear dynamical processes. Evaluations with 28 musicians demonstrated the potential for these processes to provoke creative surprises, and support exploration without a corresponding loss of control. A subsequent ethnographically-informed study with 24 musicians linked these findings to a mode of engagement which we term 'edge-like interaction'. Edge-like interactions draw on the complex, unpredictable behaviours found in nonlinear dynamical processes close to critical thresholds, facilitating creative exploration. The two complementary studies provide evidence both for the existing importance of nonlinear dynamical processes in musical interactions with acoustic interactions, and their potential for deployment in the development of new creative digital technologies, musical or otherwise.
The goal of this project was to find ways of using artificial intelligence to encourage and aid m... more The goal of this project was to find ways of using artificial intelligence to encourage and aid music composition by musical novices, particularly those without traditional musical skills. Two complementary approaches are presented. In the first approach, Harmony Space, a new kind of direct manipulation tool for music, is described. Two recent cognitive theories of harmony demonstrated how to design Harmony Space, with the expressivity to allow novices to sketch, analyze, modify, and compose harmonic sequences simply and clearly by moving two-dimensional patterns on a computer screen linked to a synthesizer. An investigation with a small number of subjects showed that Harmony Space considerably reduces the prerequisites required for novices to learn about, sketch, analyze, and experiment with harmony--activities that would normally be very difficult without considerable theoretical knowledge or instrumental skill. The second approach described is a knowledge-based tutoring system to help novices use an interface to compose chord sequences; this system is in the developmental stage. It is argued that traditional, remedial intelligent tutoring systems approaches were inadequate for tutoring in domains that require open-ended thinking. The foundation of a new approach is developed based on the exploration and transformation of case studies described in terms of chunks, styles, and plans. This approach draws on the characterization of creativity of . Programs have been implemented to illustrate the feasibility of key parts of the new approach. (Author/DB)
Interaction design for domains that involve complex abstractions can prove challenging. This prob... more Interaction design for domains that involve complex abstractions can prove challenging. This problem is particularly acute in domains where the intricate nature of domain-specific knowledge can be difficult for even the most experienced expert to conceptualise or articulate. One promising solution to the problem of representing complex domain abstractions involves the use of conceptual metaphors. Previous applications of conceptual metaphors to abstract domains have yielded encouraging results. However, the design of appropriate methods for eliciting conceptual metaphors for the purposes of informing interaction design remains an open question. In this paper, we report on a series of studies carried out to elicit conceptual metaphors from domain experts, using music as a case study, reflecting on the benefits and drawbacks of each approach.
Historically, audiences have had various ways to participate in live music performances, includin... more Historically, audiences have had various ways to participate in live music performances, including clapping, dancing, swaying, whistling, and singing. More recently, mobile and wireless devices, such as smartphones have opened up powerful new opportunities for audience participation. However, design for technology-mediated audience participation (TMAP) can be challenging: musicians and audiences have different demands, as does the coherence of the music, and group needs can vary widely. Thus, effective TMAP design requires the balancing of knowledge from diverse perspectives and must take into account the needs of diverse roles in creating and supporting performances. This chapter focuses on the process of creating and evaluating a set of design cards to support the interaction design and evaluation of TMAP systems. The cards are based on a previously created descriptive framework for supporting interaction design and evaluation in this challenging area. We discuss the conception and development of the TMAP design cards in some detail, and present an empirical study to evaluate their practical usefulness. Particular attention is paid to the ability of the cards to support finding ideas, changing ideas, and examining ideas from different perspectives.
Zenodo (CERN European Organization for Nuclear Research), Sep 27, 2022
One of the key missions of the Polifonia project is to build and interlink knowledge graphs of Eu... more One of the key missions of the Polifonia project is to build and interlink knowledge graphs of European musical cultural heritage. These interconnected music knowledge graphs act as a data backbone for the project. This deliverable is centred around one of the ten Polifonia pilots: INTERLINK. The INTERLINK pilot is not bound to a specific institutional dataset, collection or use case; but rather summarises the common needs of all pilots of representing their data as music knowledge graphs, and finding meaningful connections among them. Existing work in the relevant areas of knowledge engineering, knowledge graph construction, and knowledge graph interlinking is promising but presents a number of pitfalls in its direct application to Polifonia from the point of view of achieving this knowledge graph building and interlinking: (a) existing knowledge engineering methodologies do not account for "in-between" knowledge graph construction processes that are not fully distributed nor local; (b) in an integrated vision of European musical cultural heritage through knowledge graphs, both musical content and its 'context' or metadata need to co-exist; and (c) no current techniques address the issue of finding missing links between different music knowledge graphs at scale, accounting for this coexistence of musical content and context in the same knowledge graph. This deliverable describes our efforts, particularly gathered in Polifonia's WP2 and Task T2.3, in addressing these challenges, delivering: (a) a scalable methodology for knowledge graph construction that is well suited for the project and sits in-between the two classic extremes of ontology engineering; (b) a music knowledge graph construction framework that leverages modules of the Polifonia Ontology Network and the SPARQL Anything tool; and (c) a music knowledge graph interlinking framework, based on both heuristics and neural network-based link prediction through music knowledge graph embeddings. The latter two are evaluated in two different use-cases, the ChoCo-KG chord annotation knowledge graph, and the MIDI2vec music knowledge graph interlinking method, with encouraging results.
The work described here forms part of a project using models of mus:..cal ideas within an artific... more The work described here forms part of a project using models of mus:..cal ideas within an artificial intelligence and education framework whose goal is to encourage and facilitate music composition by novices. Formal knowledge of the domain (popular music) is too incomplete and fragmented to support a traditional expert-based tutor for precisely constrained tasks with clear-cut rules. Instead the proposed system will try to construct teaching plans on the basis of what novices can do, and equally importantly, what they like, in order to help them find paths to personally important goals. As well as an attempt to tackle the stringent demands of the particular domain, the research is an exploration of whether it is possible to tutor effectively with incomplete knowledge in a complex domain. A 20-item reference list is included.
A key challenge in the design of Virtual Musical instruments (VMIs) is finding expressive, playab... more A key challenge in the design of Virtual Musical instruments (VMIs) is finding expressive, playable, learnable mappings from gesture to sound that progressively reward practice by performers. Designing such mappings can be particularly demanding in the case of non-contact musical instruments, where physical cues can be scarce. Unaided intuition works well for many instrument designers, but others may find design and evaluation heuristics useful when creating new VMIs. In this paper we gather existing criteria from the literature to assemble a simple set of design and evaluation heuristics that we dub articulacy. This paper presents a design case study in which an expressive non-contact finger-tracking VMI, Sound Spheres, is designed and evaluated with the support of the articulacy heuristics. The case study explores the extent to which articulacy usefully informs the design of a non-contact VMI, and we reflect on the usefulness or otherwise of heuristic approaches in this context.
This is the introductory chapter of a book dedicated to new research in, and emerging new underst... more This is the introductory chapter of a book dedicated to new research in, and emerging new understandings of, music and human-computer interactionknown for short as music interaction. Music interaction research plays a key role in innovative approaches to diverse musical activities, including performance, composition, education, analysis, production and collaborative music making. Music interaction is pivotal in new research directions in a range of activities, including audience participation, interaction between music and dancers, tools for algorithmic music, music video games, audio games, turntablism and live coding. More generally, music provides a powerful source of challenges and new ideas for human-computer interaction (HCI). This introductory chapter reviews the relationship between music and human-computer interaction and outlines research themes and issues that emerge from the collected work of researchers and practitioners in this book.
Uploads
Papers by Simon Holland