Audio Descriptors

As of today, most movie recommendation services base their recommendations on collaborative filtering (CF) and/or content-based filtering (CBF) models that use metadata (e.g., genre or cast). In most video-on-demand and streaming services, however, new movies and TV series are continuously added. CF models are unable to make predictions in such a scenario, since the newly added videos lack interactions – a problem technically known as new item cold start (CS). Currently, the most common approach to this problem is to switch to a purely CBF method, usually by exploiting textual metadata. This approach is known to have lower accuracy than CF because it ignores useful collaborative information and relies on human-generated textual metadata, which are expensive to collect and often prone to errors. User-generated content, such as tags, can also be rare or absent in CS situations. In this paper, we introduce a new movie recommender system that addresses the new item problem in the movie domain by (i) integrating state-of-the-art audio and visual descriptors, which can be automatically extracted from video content and constitute what we call the movie genome; (ii) exploiting an effective data fusion method named canonical correlation analysis (CCA), which was successfully tested in our previous works, to better exploit complementary information between different modalities; (iii) proposing a two-step hybrid approach which trains a CF model on warm items (items with interactions) and leverages the learned model on the movie genome to recommend cold items (items without interactions). Experimental validation is carried out using a system-centric study on a large-scale, real-world movie recommendation dataset both in an absolute cold start and in a cold to warm transition; and a user-centric online experiment measuring different subjective aspects, such as satisfaction and diversity. Results show the benefits of this approach compared to existing approaches.

Resumo: Farben Op. 16 no3 é uma obra em que Arnold Schoenberg implementa seu modelo de melodia de timbres (Klangfarbenmelodie), modelo este que talvez nunca tenha sido repetido da mesma maneira em suas demais obras. Neste artigo investigamos os processos de composição e percepção do timbre nesta obra. Como referencial teórico abordamos a busca estética do compositor por uma relação entre cores e sons, fundamentada nas ideias Goethe (1840) e Helmholtz (1924, 1954). Baseamo-nos também em teorias sobre a
percepção da altura por fusão de parciais e sobre a fusão de diferentes timbres por jitter. Sobre Farben realizamos uma análise da partitura baseada principalmente em trabalhos prévios de Förtig (1969) e Burkhart (1972). A seguir partimos para uma investigação psicoacústica da percepção do timbre, para tanto aplicamos ao áudio da obra os descritores croma, centroide, sharpness, loudness e inarmonicidade (através do programa Sonic Visualizer), os quais nos forneceram dados para realizar interpretações individuais destes parâmetros, bem como nos permitindo estabelecer relações entre eles.

Abstract: Farben Op. 16 no3 is a work in which Arnold Schoenberg implements his Klangfarbenmelodie model, which perhaps had never been repeated in the same way in his other works. In this article we explore the processes of timbre composition and perception of this work. As a theoretical basis we address the aesthetic search of the composer for a relation between colors and sounds, based on the ideas of Goethe (1840) and Helmholtz (1924, 1954). We also mention theories about pitch perception through harmonic fusion and the fusion of different timbres by jitter. On Farben, we accomplished an analysis of the score, especially considering previous analyses of Förtig (1969) and Burkhart (1972). Furthermore, we focused on the psychoacoustic investigation of timbre perception. For this purpose, audio descriptors such as chroma, spectral centroid, sharpness, loudness and inharmonicity were applied to the audio through Sonic Visualizer software. These descriptors provided us information to perform individual interpretations of these parameters and also to establish relationships between them.
Keywords: Arnold Schoenberg; Klangfarbenmelodie (timbre melody); Farben; timbre perception; timbre fusion; pitch perception.

Neste artigo, pertencente à pesquisa sobre a elaboração de um modelo para a análise da música eletroacústica mista a partir de descritores de áudio, realizamos a análise de Ressonâncias (2015) para piano e sons eletrônicos de Jônatas Manzolli. Como fundamentação teórica, apresentamos uma abordagem sobre as Séries e a Transformada de Fourier, operação que serve de base para os processamentos eletroacústicos empregados na composição e para as análises realizadas a partir dos descritores. As hipóteses a serem verificadas dizem respeito à maleabilidade dos processamentos eletroacústicos empregados na peça, que podem fornecer resultados distintos em interpretações diferentes, além da existência de uma tendência de maior similaridade espectral no processo composicional realizado, em que o nível de variação entre duas interpretações é maior nas seções iniciais e menor nas seções finais. Como metodologia, analisamos as características espectrais dos sons compostos em suporte fixo e comparamos o timbre produzido (amálgama dos sons instrumentais e eletroacústicos) em duas interpretações diferentes, a partir dos dados obtidos nas análises. Concluímos discutindo dois aspectos: (1) sobre o nível de maleabilidade sonora atingido pelo compositor na estruturação do processo criativo, e (2) a análise desenvolvida aponta para a impossibilidade da notação musical tradicional de representar a completude dos resultados sonoros emergentes, que decorrem da interação entre os sons instrumentais e eletroacústicos. Ou seja, o modelo de análise estudado justifica a sua aplicabilidade no contexto da música eletroacústica mista atual, principalmente quando foca a fusão dinâmica da parte instrumental com o suporte fixo e/ou com o tratamento em tempo real.

From Montserrat to Piano Resonances: An Analysis with Audio Descriptors

Abstract: As part of a research study of an audio descriptors model for the analysis of live-electronic music, this article addresses Ressonâncias (2015) for piano and electronic sounds by Jônatas Manzolli. As a theoretical basis, we approach the Fourier Series and Transform, an operation that serves as a basis for the electroacoustic processes employed in the composition and for the analyses performed from descriptors. The hypotheses to be verified are related to the malleability of the electroacoustic processes employed in the piece, which can deliver specific results for different performances, besides the existence of a tendency of greater spectral similarity in the compositional process, where the level of variation between two performances is higher in the initial sections and lower in the final sections. As a methodology, we analyze the spectral features of the fixed tape sounds and compare the timbre produced (the amalgam of instrumental and electroacoustic sounds) in two different performances from the data obtained from the analysis. We conclude by discussing two aspects: (1) the level of sonic malleability reached by the composer in structuring his creative process, and (2) the analysis that emphasizes the impossibility of traditional music notation to represent the plentitude of the emerging sound resulting from the interaction between instrumental and electroacoustic sounds. In other words, the applicability of the analytical model studied is relevant in the context of current live-electronic music, mainly when it addresses the dynamic fusion of the instrumental part with tape sounds and/or real-time treatments.