VISUALIZING AND CONTROLLING SOUND WITH GRAPHICAL INTERFACES

2019

This paper describes Sonifyd, a sonification driven multimedia and audiovisual environment based on color-sound conversion for real-time manipulation. Sonifyd scans graphics horizontally or vertically from a scan line, generates sound and determines timbre according to its own additive synthesis based color-to-sound mapping. Color and sound relationships are fixed as default, but they can be organic for more tonal flexibility. Within this ecosystem, flexible timbre changes will be discovered by Sonifyd. The scan line is invisible, but Sonifyd provides another display that represents the scanning process in the form of dynamic imagery representation. The primary goal of this project is to be a functioning tool for a new kind of visual music, graphic sonification research and to further provide a synesthetic metaphor for audiences/users in the context of an art installation and audiovisual performance. The later section is a discussion about limitations that I have encountered: using ...

2021

This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY

Proceedings of Sound and Music Computing Conference 2012, 2012

This paper compares two multimodal net art projects, AVOL and AV Clash, by the author and André Carrilho (under the name Video Jack). Their objective is to create projects enabling integrated audiovisual expression that are flexible, easy to use, playful and engaging to experience. The projects are contextualized with related works. The methodology for the research is presented, with an emphasis on experience-focused Human-Computer Inter- action (HCI) perspectives. The comparative evaluation of the projects focuses on the analysis of the answers to an online questionnaire. AVOL and AV Clash have adopted an Interactive AudioVisual Objects (IAVO) approach, which is a major contribution from these projects, con- sisting of the integration of sound, audio visualization and Graphical User Interface (GUI). Strengths and weaknesses detected in the projects are analysed. Generic conclusions are discussed, mainly regarding simplicity and harmony versus complexity and serendipity in audiovis- ual projects. Finally, paths for future development are presented.

International Journal of Human-Computer Studies, 2015

IEEE Multimedia, 2005

T he research field of sonification, a subset of the topic of auditory display, has developed rapidly in recent decades. It brings together interests from the areas of data mining, exploratory data analysis, human-computer interfaces, and computer music. Sonification presents information by using sound (particularly nonspeech), so that the user of an auditory display obtains a deeper understanding of the data or processes under investigation by listening. 1 We define interactive sonification as the use of sound within a tightly closed human-computer interface where the auditory signal provides information about data under analysis, or about the interaction itself, which is useful for refining the activity.

GENERATIVE VISUALISATION BASED ON SOUND

This paper presents the creation of a prototype system for audio analysis and visualization. To build a sound visualization system that can be truly and intuitively perceived by the human brain is a great chal-lenge. Having such system can aid people who have hearing problems and help them enter the collective culture of music and society. Using such utility humans can further expand their knowledge and understanding of the way liv-ing beings communicate. The first part of the prototype is aimed at using specif-ic techniques of sound analysis to extract data in real time. The second part focuses on the visualization of the collected data in a way that can be visually perceived by the human eye.

2016

The present research sought to investigate the correspondence between auditory and visual feature dimensions and to utilise this knowledge in order to inform the design of audiovisual mappings for visual control of sound synthesis. The first stage of the research involved the design and implementation of Morpheme, a novel interface for interaction with corpus-based concatenative synthesis. Morpheme uses sketching as a model for interaction between the user and the computer. The purpose of the system is to facilitate the expression of sound design ideas by describing the qualities of the sound to be synthesised in visual terms, using a set of perceptually meaningful audiovisual feature associations. The second stage of the research involved the preparation of two multidimensional mappings for the association between auditory and visual dimensions. The third stage of this research involved the evaluation of the AudioVisual (A/V) mappings and of Morpheme's user interface. The evaluation comprised two controlled experiments, an online study and a user study. Our findings suggest that the strength of the perceived correspondence between the A/V associations prevails over the timbre characteristics of the sounds used to render the complementary polar features. Hence, the empirical evidence gathered by previous research is generalizable/ applicable to different contexts and the overall dimensionality of the sound used to render should not have a very significant effect on the comprehensibility and usability of an A/V mapping. However, the findings of the present research also show that there is a non-linear interaction between the harmonicity of the corpus and the perceived correspondence of the audiovisual associations. For example, strongly correlated cross-modal cues such as size-loudness or vertical position-pitch are affected less by the harmonicity of the audio corpus in comparison to weaker correlated dimensions (e.g. texture granularity-sound dissonance). No significant differences were revealed as a result of musical/audio training. The third study consisted of an evaluation of Morpheme's user interface were participants were asked to use the system to design a sound for a given video footage. The usability of the system was found to be satisfactory. An interface for drawing visual queries was developed for high level control of the retrieval and signal processing algorithms of concatenative sound synthesis. This thesis elaborates on previous research findings and proposes two methods for empirically driven validation of audiovisual mappings for sound synthesis. These methods could be applied to a wide range of contexts in order to inform the design of cognitively useful multi-modal interfaces and representation and rendering of multimodal data. Moreover this research contributes to the broader understanding of multimodal perception by gathering empirical evidence about the Augoustinos Tsiros Thesis ii correspondence between auditory and visual feature dimensions and by investigating which factors affect the perceived congruency between aural and visual structures.

MUSICULT 16 PROCEEDINGS, 2016

Technology of sound synthesis from a light is a combination of hundreds of years of research in mathematics, optics, electronics, art theory, psychology and communications. Many significant European culture leaders associated image and sound theoretically (Pythagoras, I. Newton, G. Sachs, etc.) while artists have tried to link the image and sound in practice (G. Arcimboldo, L. B. Castel, F. E. Kastner, P. O. Runge, A. Scriabin, N. Rimsky-Korsakov, etc.). Various tools and technologies were used to connect or translate images to sounds (gravicembalo, harpsicord, pyrophone, etc.). In twentieth century graphical sounds were explored in radio, film, video and television almost in the Soviet Union, Germany, France, United Kingdom and the United States of America. Nowadays, significant tools and achievements resulting from the method of graphical sound are available to audience in specialized archives and educational institutions. In twenty-first century, tools and technologies for synthesizing sound from light comprise a set of software solutions for different operational systems including applications for mobile devices. They are still in use in radio, film, video, television, new media art and industry and in masterpieces as well. Software solutions are simple and enable even beginners reach original and valuable results. ČIPke (Ljubljana, Slovenia), a non-profit initiative for researching the conditions for women who are active in the context of science, technology and media art, develops and organizes practical educational workshops on graphical sound synthesis. Small groups of women learn about theoretical background of graphical sound and they practice. At the end of the workshop they are able to apply the knowledge in practice and implement graphical sound to their projects. Significant amount of sounds is produced for videos, installations, visualizations and the group of authors regularly participate in art exhibitions and screenings.

Personal and Ubiquitous Computing, 2020

Graphical interpolation systems provide a simple mechanism for the control of sound synthesis by providing a level of abstraction above the engine parameters, allowing users to explore different sounds without awareness of the underlying details. Typically, a graphical interpolator presents the user with a two-dimensional pane where a number of synthesizer presets, each representing a collection of synthesis parameter values, can be located. Moving an interpolation cursor within the pane results in the calculation of new parameter values, based on its position, the relative locations of the presets, and the mathematical interpolation function, thus generating new sounds. These systems supply users with two sensory modalities in the form of sonic output and visual feedback from the interface. A number of graphical interpolator systems have been developed over the years, with a variety of user-interface designs, but few have been subject to formal user evaluation. Our testing studied ...

AudioMostly 2022

I designed two sonification platforms designed for visual/auditory communication design studies and audiovisual art. The purpose of this study was to examine whether test participants can associate visuals and sound without any prior training and sonification approaches in this paper can be utilized as an interactive musical expression. The platform for the communication design study was developed first and the artistic audiovisual platform with the same sonification methodology followed next. In this paper, I introduce the (former) sonification platform designed for the image-to-sound association studies, their sonification methodologies, and present the study results. The object-oriented sonification method that I newly developed describes each shape sonically. The five imagesound association studies were conducted to see whether people can successfully associate sounds and fundamental shapes (i.e., a circle, a triangle, a square, lines, curves, and other custom shapes). Regardless of age and educational background, the correct answer rate was high.