580 California St., Suite 400
San Francisco, CA, 94104
Key research themes
1. How can scalable content-based search frameworks be designed to efficiently retrieve music documents in diverse digital encodings?
This research area focuses on developing scalable and generic frameworks capable of content-based search across large music collections encoded in heterogeneous digital formats (such as audio recordings, symbolic scores in XML, MIDI, or specialized encodings). The challenge lies in representing complex, multifaceted musical content independently from encoding specifics and integrating feature extraction, indexing, and search/ranking within scalable architectures (e.g., leveraging text search engine principles). Such frameworks are crucial for enabling fast, sublinear search in massive music libraries, going beyond metadata-based retrieval to support queries on rich musical content elements (e.g., melodic patterns in specific parts).
Key finding: This paper introduces a general framework that decouples music content representation from encoding using a Music Content Model (MCM), supports extendible feature extraction, and integrates search/index/rank operations within... Read more
Key finding: The MiDiLiB project surveyed various content-based retrieval techniques for both symbolic and audio-based music, emphasizing the lifecycle challenges including digitization, format choice, content analysis, and indexing. The... Read more
Key finding: This review presents CB-MIR tasks focusing on extracting musical meta-information (genre, artist, vocal segmentation) directly from audio content without requiring manual annotations. It articulates the need for scalable,... Read more
2. What audio feature extraction and representation methods enable robust retrieval and classification of music from raw audio signals?
This theme addresses approaches that extract meaningful features directly from raw audio signals to support retrieval tasks such as genre classification, query-by-example, segmentation, and audio thumbnailing. The research emphasizes the design and evaluation of spectral, timbral, temporal, and statistical features (e.g., MFCCs, LPC, FFT) and their combinations, as well as advanced visualization techniques (e.g., TimbreGrams) and classifications. The challenge is to effectively capture musical characteristics from audio signals under conditions of noise, variability, and non-symbolic formats, surpassing transcription limitations.
Key finding: This paper develops an extensible audio retrieval system focused on feature-based audio analysis at small time scales (FFT, LPC, MFCC) and supports audio classification, segmentation, retrieval, and audio thumbnailing without... Read more
Key finding: The authors propose novel user interfaces that extend beyond traditional query-by-example for audio retrieval by enabling users to synthesize or combine new audio queries based on content-derived parameters. This relies on... Read more
Key finding: This study presents the merger of two prototype systems (Sonic Browser and MARSYAS), combining audio analysis, classification, segmentation, and novel visualization (TimbreSpaces) to provide a flexible 2D/2.5D audio browsing... Read more
Key finding: This paper systematically investigates integration of short-time audio features (e.g., MFCCs) over longer time scales through statistical approaches (mean, variance) and proposes using autoregressive models for feature... Read more
3. How can melodic similarity and structural patterns be leveraged to identify cover versions and structural relationships between music recordings?
This theme investigates methodologies for detecting melodic similarity and structural characteristics to identify cover songs, versions, and versions sharing underlying structure despite different performances. It builds on symbolic representation extraction (e.g., sung notes) and self-similarity matrices to capture deeper musical relations, accommodating transformations like tempo variation, transposition, and accompaniment. These approaches enable content-based retrieval that transcends metadata and superficial audio matching, crucial for music version identification, copyright management, and indexing live or polyphonic recordings.
Key finding: The paper develops a method for identifying cover versions by extracting and comparing main vocal melodies from polyphonic audio, using preprocessing to remove accompaniment and tempo normalization. Experiments on 594... Read more
Key finding: This work introduces structure fingerprints derived from self-similarity matrices as compact descriptors of a recording's structural properties, enabling comparison without complex alignment. Using kernel density estimation... Read more
Key finding: The authors evaluate various time series representations of sung queries (pitch contours, sequences of notes, and novel pitch histogram sequences) combined with dynamic time warping variants for alignment. They show that... Read more