Mutation testing is a well-established technique for assessing a test suite's quality by injectin... more Mutation testing is a well-established technique for assessing a test suite's quality by injecting artificial faults into production code. In recent years, mutation testing has been extended to machine learning (ML) systems, and deep learning (DL) in particular; researchers have proposed approaches, tools, and statistically sound heuristics to determine whether mutants in DL systems are killed or not. However, as we will argue in this work, questions can be raised to what extent currently used mutation testing techniques in DL are actually in line with the classical interpretation of mutation testing. We observe that ML model development resembles a test-driven development (TDD) process, in which a training algorithm ('programmer') generates a model (program) that fits the data points (test data) to labels (implicit assertions), up to a certain threshold. However, considering proposed mutation testing techniques for ML systems under this TDD metaphor, in current approaches, the distinction between production and test code is blurry, and the realism of mutation operators can be challenged. We also consider the fundamental hypotheses underlying classical mutation testing: the competent programmer hypothesis and coupling effect hypothesis. As we will illustrate, these hypotheses do not trivially translate to ML system development, and more conscious and explicit scoping and concept mapping will be needed to truly draw parallels. Based on our observations, we propose several action points for better alignment of mutation testing techniques for ML with paradigms and vocabularies of classical mutation testing.
International Symposium/Conference on Music Information Retrieval, Oct 11, 2020
Psychology research has shown that song lyrics are a rich source of data, yet they are often over... more Psychology research has shown that song lyrics are a rich source of data, yet they are often overlooked in the field of MIR compared to audio. In this paper, we provide an initial assessment of the usefulness of features drawn from lyrics for various fields, such as MIR and Music Psychology. To do so, we assess the performance of lyric-based text features on 3 MIR tasks, in comparison to audio features. Specifically, we draw sets of text features from the field of Natural Language Processing and Psychology. Further, we estimate their effect on performance while statistically controlling for the effect of audio features, by using a hierarchical regression statistical model. Lyric-based features show a small but statistically significant effect, that anticipates further research. Implications and directions for future studies are discussed.
Bedankt voor het downloaden van dit artikel. De artikelen uit de (online)tijdschriften van Uitgev... more Bedankt voor het downloaden van dit artikel. De artikelen uit de (online)tijdschriften van Uitgeverij Boom zijn auteursrechtelijk beschermd. U kunt er natuurlijk uit citeren (voorzien van een bronvermelding) maar voor reproductie in welke vorm dan ook moet toestemming aan de uitgever worden gevraagd. Behoudens de in of krachtens de Auteurswet van 1912 gestelde uitzonderingen mag niets uit deze uitgave worden verveelvoudigd, opgeslagen in een geautomatiseerd gegevensbestand, of openbaar gemaakt, in enige vorm of op enige wijze, hetzij elektronisch, mechanisch door fotokopieën, opnamen of enig andere manier, zonder voorafgaande schriftelijke toestemming van de uitgever. Voor zover het maken van kopieën uit deze uitgave is toegestaan op grond van artikelen 16h t/m 16m Auteurswet 1912 jo. Besluit van 27 november 2002, Stb 575, dient men de daarvoor wettelijk verschuldigde vergoeding te voldoen aan de Stichting Reprorecht te Hoofddorp (postbus 3060, 2130 KB, www.reprorecht.nl) of contact op te nemen met de uitgever voor het treffen van een rechtstreekse regeling in de zin van art. 16l, vijfde lid, Auteurswet 1912. Voor het overnemen van gedeelte(n) uit deze uitgave in bloemlezingen, readers en andere compilatiewerken (artikel 16, Auteurswet 1912) kan men zich wenden tot de Stichting PRO (Stichting Publicatie-en Reproductierechten, postbus 3060, 2130 KB Hoofddorp, www.cedar.nl/pro).
Prior research from the field of music psychology has suggested that there are factors common to ... more Prior research from the field of music psychology has suggested that there are factors common to music preference beyond individual genres. Specifically, research has shown that self-reported ratings of preference for individual musical genres can be reduced to 4 or 5 dimensions, which in turn have been shown to correlate to relevant psychological constructs, such as personality. However, the number of dimensions emerging from multiple studies has varied despite the care taken in conducting such research. Data-driven approaches offer opportunities to further this line of research with actual listening data, at a scale and scope surpassing that of traditional psychological studies. Although listening data can be considered more direct and comprehensive evidence of listening preference, transforming this data into meaningful measurements is non-trivial. In the current paper, we report on investigations seeking to find interpretable underlying dimensions of music taste, using implicit large-scale listening data. Offering a critical reflection on potential researchers' degrees of freedom, we adopt an explicit systematic approach, investigating the impact of varying different parameters, analysis, and normalization techniques. More precisely, we consider various ways to extract listening preference information from two large, openly available datasets of music listening behavior, making use of principal component analysis and variational autoencoders to extract potential underlying dimensions. Results and implications are discussed in light of prior psychological theory, and the potential of user listening data to further research on music preference.
Deep neural networks have frequently been used to directly learn representations useful for a giv... more Deep neural networks have frequently been used to directly learn representations useful for a given task from raw input data. In terms of overall performance metrics, machine learning solutions employing deep representations frequently have been reported to greatly outperform those using hand-crafted feature representations. At the same time, they may pick up on aspects that are predominant in the data, yet not actually meaningful or interpretable. In this paper, we therefore propose a systematic way to test the trustworthiness of deep music representations, considering musical semantics. The underlying assumption is that in case a deep representation is to be trusted, distance consistency between known related points should be maintained both in the input audio space and corresponding latent deep space. We generate known related points through semantically meaningful transformations, both considering imperceptible and graver transformations. Then, we examine within-and between-space distance consistencies, both considering audio space and latent embedded space, the latter either being a result of a conventional feature extractor or a deep encoder. We illustrate how our method, as a complement to task-specific performance, provides interpretable insight into what a network may have captured from training data signals.
Over the past millennia, music has actively been performed and listened to by mankind, thus also ... more Over the past millennia, music has actively been performed and listened to by mankind, thus also playing an important role in establishing sociocultural identities that have evolved over time. In parallel, for many centuries, newspapers played an important role in informing society on a regular and frequent basis on topics noteworthy at that time. Therefore, in retrospect, these newspapers offer windows into historic topics of sociocultural significance, including cultural and musical life. Thanks to ongoing digitization efforts, large-scale newspaper corpora now have become broadly available and accessible. Taking the digitized historical newspaper collection of the National Library of The Netherlands as an example, in this paper, we discuss how considering music-related mentionings in newspapers can enable potential new research directions and questions. We discuss open syntactic and semantic data-related technical challenges when analyzing music-related mentionings in digitized historical newspaper collections. Finally, we discuss how successful detection of music-related mentionings can also benefit engagement of non-scholarly end users, concluding with an invitation to the interdisciplinary research community to actively contribute to the given use case. CCS CONCEPTS • Applied computing → Arts and humanities; Digital libraries and archives; Law, social and behavioral sciences; • Information systems → Web searching and information discovery; • Social and professional topics → Cultural characteristics;
Suranga Nanayakkara is an Assistant Professor from the Engineering Product Development Pillar at ... more Suranga Nanayakkara is an Assistant Professor from the Engineering Product Development Pillar at Singapore University of Technology and Design (SUTD). Before joining SUTD, Suranga was a Postdoctoral Associate at the Fluid Interfaces group, MIT Media Lab. He received his PhD in 2010 and BEng in 2005 from the National University of Singapore. In 2011, he founded the "Augmented Human Lab" (www.ahlab.org) to explore ways of creating 'enabling' human-computer interfaces to enhance the sensory and cognitive abilities of humans. With publications in prestigious conferences, demonstrations, patents, media coverage and real-world deployments, Suranga has demonstrated the potential of advancing the state-of-the art in Assistive Human Computer Interfaces. For the totality and breadth of achievements, Suranga has been recognized with many awards, including young inventor under 35 (TR35 award) in the Asia Pacific region by MIT TechReview, Ten Outstanding Yong Professionals (TOYP) by JCI Sri Lanka and INK Fellow 2016.
There is 'multi' in multimedia. Every day, an increasing amount of extremely diverse mult... more There is 'multi' in multimedia. Every day, an increasing amount of extremely diverse multimedia content has meaning and purpose to an increasing amount of extremely diverse human users, under extremely diverse use cases. As multimedia professionals, we work in an extremely diverse set of focus areas to enable this, ranging from systems aspects to user factors, which each have their own methodologies and related communities outside of the multimedia field.
In this workshop, we will interactively and jointly investigate with the community how intrinsic ... more In this workshop, we will interactively and jointly investigate with the community how intrinsic creative and artistic values, technology and society are critical to one another, and how our present-day urban and digital societies can demonstrably be enriched and advanced by effective connections between these all. The workshop will include interactive discussions, as well as a hands-on outdoors making session.
This paper provides a three-step framework to predict user assessment of the suitability of movie... more This paper provides a three-step framework to predict user assessment of the suitability of movies for an inflight viewing context. For this, we employed classifier stacking strategies. First of all, using the different modalities of training data, twenty-one classifiers were trained together with a feature selection algorithm. Final predictions were then obtained by applying three classifier stacking strategies. Our results reveal that different stacking strategies lead to different evaluation results. A considerable improvement can be found for the F1-score when using the label stacking strategy.
Mass Media Musical Meaning: Opportunities from the Collaborative Web
In the digital domain, music is usually studied from a positivist viewpoint, focusing on general ... more In the digital domain, music is usually studied from a positivist viewpoint, focusing on general ‘objective’ music descriptors. In this work, we strive to put music in a more social and cultural context, looking into ways to unify data analysis methods with thoughts from the humanities on musical meaning and significance. More specifically, we investigate whether information in collaborative web resources on movie plot narratives and folksonomic song tags is capable of revealing common associations between these two. Reported initial findings suggest this is indeed the case, which opens opportunities for further work in this area, cross-disciplinary collaborations, and novel contextually oriented music information retrieval application scenarios.
, after 'Floating Apples' by Sandy (Nelson) Maynard. A full attribution and further notes can be ... more , after 'Floating Apples' by Sandy (Nelson) Maynard. A full attribution and further notes can be found at the back of this thesis.
A study is presented of the cover song retrieval problem, in which multiple performances of the s... more A study is presented of the cover song retrieval problem, in which multiple performances of the same musical work are sought. The problem is considered in the context of content-based audio retrieval. In order to gain more insight into the current state-of-the art in cover song retrieval research, several existing approaches to cover song retrieval are studied in a modular way. Modules from different approaches are systematically recombined and each resulting combination is tested on several designated datasets. The modules consider feature extraction and representation, (dis)similarity assessment, averaging factors for data reduction and feature normalization techniques and have been chosen such that they reflect general and independent system decisions. Datasets have been constructed such that they each pose a specific and known subset of the broad range of cover song types and similarity challenges. For the experiments that are carried out, we depart from cover song system combinations that use the conventional approach of representing songs as absolute chroma vectors over time. Additionally, we transform these representations into a statistical meta-representation and also study the influences of using relative first-order timedifferential information. The benefit of this modular approach to cover song retrieval is shown in the obtained experimental results. In experiments with the conventional representations, the (dis)similarity assessment method and averaging factor were identified as the components that currently cause the largest performance differences between system combinations. On opus retrieval test cases, in which several performers play exactly the same score, the most successful system combinations achieve near-perfect results. In these opus retrieval cases, the statistical meta-representation is able to achieve very good performance results as well, while being computationally much less complex than the local alignment techniques needed to successfully assess (dis)similarity between conventional representations. For the statistical meta-representation, we identified that an important influence on performance results was the choice of feature representation to transform. Using time-differential information has advantages when combined with absolute information over time, but current experiments with pure time-differential information yielded inferior results in performance compared to experiments with absolute information. We present our work by discussing the cover song retrieval problem in a topdown way, starting from general musical and technical issues posed. Subsequently, we Contents Preface iii Contents v List of Figures vii List of Tables ix List of Common Abbreviations and Mathematical Symbols xi List of Common Abbreviations and Mathematical Symbols Abbreviations AUC Area Under the Curve CC Cross-Correlation CIF Chromagram from Instantaneous Frequency DFT Discrete Fourier Transform DPLA Dynamic Programming Local Alignment FFT Fast Fourier Transform HPCP Harmonic Pitch Class Profile IF Instantaneous Frequency MAP Mean of Average Precisions MIR Music Information Retrieval MIREX Music Information Retrieval EXchange MR1st Mean Rank of 1st correctly identified cover OTI Optimal Transposition Index PCP Pitch Class Profile ROC Receiver Operating Characteristic Mathematical Symbols ∆x first-order time-differential chroma representation over time xi LIST OF TABLES Φ chroma fingerprint (covariance matrix) x chroma representation over time (chromagram) f frequency r total number of chroma bins used in one octave tr amount of transposition xii
Recommender systems can be useful in group settings, e.g. when choosing a movie to watch with a g... more Recommender systems can be useful in group settings, e.g. when choosing a movie to watch with a group. However, while considerable research in group recommendation has been performed, we still lack truly ecological datasets on group recommendations in real life consumption scenarios. Much of the existing work considers hypothetical consumption scenarios, and commonly, individual ratings are aggregated, but no actual group consumption takes place in which situational differences per group are taken into account. In this paper, we outline a vision for acquiring more realistic and ecological group consumption data, based on a crowdsourcing application that will acquire individual ratings per group consumption event. We discuss various design decisions that will allow us to gather these ratings effectively from a large group of people, and demonstrate and evaluate the viability of our approach towards reaching group consensus through rating session simulations.
Humans have the capacity to invent novel ideas and to create new artifacts that affect the surrou... more Humans have the capacity to invent novel ideas and to create new artifacts that affect the surrounding environment. However, it is unclear how this capacity emerges and develops in biological systems. This paper presents an empirical study which investigates the development of novelty-based cognitive processes in the context of unstructured music-making activities in early childhood. We used principles of intuitive theories of emergence, the paradigm of overlapping waves of mechanisms of change and theories of music cognitive development to theoretically conceptualize the developmental process in the specific context. We applied the methodological principles of microgenetic analysis for the development of an annotation scheme of micro-behaviors, which correspond to a set of cognitive processes. We took into consideration child's behavioral manifestations of music-induced affective engagement, as an indicator of intrinsic motivation. Our results suggest that the process of transition from spontaneous towards deliberate actions develops through exploratory actions, evaluation of the outcomes, reasoning and planning. The structure of these actions appears in the form of dynamic overlapping waves rather than in a linear or iterative manner. Additionally, our results indicate that children in early years make use of the affordances of the provided tools to scaffold their transition from concrete visual representation of sonic features towards abstract musical thinking, which suggests that musical development appears with the generative tension between action and symbol. Implications and future work are discussed regarding the development of intelligent robotic systems for user adaptive scaffolding of the observed mechanisms of change.
Music frequently occurs as an important reinforcing and meaning-creating element in multimodal hu... more Music frequently occurs as an important reinforcing and meaning-creating element in multimodal human experiences. This way, cross-modal connotative associations are established, which are actively exploited in professional multimedia productions. A lay user who wants to use music in a similar way may have a result in mind, but may lack the right musical vocabulary to express the corresponding information need. However, if the connotative associations between music and visual narrative are strong enough, characterizations of music in terms of a narrative multimedia context can be envisioned. In this article, we present the outcomes of a user study considering this problem. Through a survey for which respondents were recruited via crowdsourcing methods, we solicited descriptions of cinematic situations for which fragments of royalty-free production music would be suitable soundtracks. As we will show, these descriptions can reliably be recognized by other respondents as belonging to the music fragments that triggered them. We do not fix any description vocabulary beforehand, but rather give respondents a lot of freedom to express their associations. From these free descriptions, common narrative elements emerge that can be generalized in terms of event structure. The insights gained this way can be used to inform new conceptual foundations for supervised methods, and to provide new perspectives on meaningful and multimedia contextaware querying, retrieval and analysis.
International Symposium/Conference on Music Information Retrieval, Aug 7, 2016
Music has been shown to have a profound effect on listeners' internal states as evidenced by neur... more Music has been shown to have a profound effect on listeners' internal states as evidenced by neuroscience research. Listeners report selecting and listening to music with specific intent, thereby using music as a tool to achieve desired psychological effects within a given context. In light of these observations, we argue that music information retrieval research must revisit the dominant assumption that listening to music is only an end unto itself. Instead, researchers should embrace the idea that music is also a technology used by listeners to achieve a specific desired internal state, given a particular set of circumstances and a desired goal. This paper focuses on listening to music in isolation (i.e., when the user listens to music by themselves with headphones) and surveys research from the fields of social psychology and neuroscience to build a case for a new line of research in music information retrieval on the ability of music to produce flow states in listeners. We argue that interdisciplinary collaboration is necessary in order to develop the understanding and techniques necessary to allow listeners to exploit the full potential of music as psychological technology.
From Water Music to ‘Underwater Music’: Multimedia Soundtrack Retrieval with Social Mass Media Resources
Lecture Notes in Computer Science, 2016
In creative media, visual imagery is often combined with music soundtracks. In the resulting arte... more In creative media, visual imagery is often combined with music soundtracks. In the resulting artefacts, the consumption of isolated music or imagery will not be the main goal, but rather the combined multimedia experience. Through frequent combination of music with non-musical information resources and the corresponding public exposure, certain types of music will get associated to certain types of non-musical contexts. As a consequence, when dealing with the problem of soundtrack retrieval for non-musical media, it would be appropriate to not only address corresponding music search engines in music-technical terms, but to also exploit typical surrounding contextual and connotative associations. In this work, we make use of this information, and present and validate a search engine framework based on collaborative and social Web resources on mass media and corresponding music usage. Making use of the SRBench dataset, we show that employing social folksonomic descriptions in search indices is effective for multimedia soundtrack retrieval.
Uploads
Papers by Cynthia Liem