Music Emotion Classification

In this thesis we focus on the automatic emotion classification of music samples. We extract a set of features from the music signal and examine their discriminatory capability using various classification techniques. Our goal is to determine the features and the classification methods that lead to the best classification of the emotion a music sample conveys. During the course of the thesis, we generated
our own dataset of annotated song samples and we examined two distinct methods of describing an emotion: using clusters consisting of various emotional states, and using a two-dimensional representation of the emotion in the Valence-Activation plane. The latter method was chosen as the most successful. We also tried other approaches of music emotion classification (MEC) as well, such as treating the song sample as an amplitude and frequency modulated (AM-FM) signal, on which we subsequently perform multiband demodulation analysis (MDA) testing various Gabor filter banks (Mel scale-based filter bank, Bark scale-based filter bank, and a number of fractional octave-based filter banks). Statistics of the Frequency Modulation Percentages (FMPs) of each band derived from the demodulation, proved to be quite successful features in the classification of emotion. Finally, we
explored other modalities besides the music sound signal itself, such as a number of features derived from the chords of the song samples, classification of the song samples' lyrics using various techniques and a brief investigation of Electroencephalogram (EEG) data generated by one of the annotators
while performing the annotation of the song samples.  Our final feature-pack included a combination of the most successful features among the ones we studied: (i) music-inspired features (features based on music theory and psychoacoustics, derived from either the sound signal or the chords of the sample), (ii) statistics of the FMPs and (iii) statistics of the Mel-frequency cepstral coefficients (MFCCs). This feature-pack proved to be more robust than its three individual components and in the end we achieved results that reached 85.7% correct classification rate in the dimension of Valence and 85.1% correct classification rate in the dimension of Activation. We finally demonstrate that by discarding training samples that are assigned a label too close to the neutral value, our results can improve even further, especially in the dimension of Activation.

This paper focuses on emotion recognition and understanding in Contemporary Western music. The study seeks to investigate the relationship between perceived emotion and musical features in the fore-mentioned musical genre. A set of 27 Contemporary music excerpts is used as stimuli to gather responses from both musicians and non-musicians which are then mapped on an emotional plane in terms of arousal and valence dimensions. Audio signal analysis techniques are applied to the corpus and a base feature set is obtained. The feature set contains characteristics ranging from low-level spectral and temporal acoustic features to high-level contextual features. The feature extraction process is discussed with particular emphasis on the interaction between acoustical and structural parameters. Statistical relations between audio features and emotional ratings from psychological experiments are systematically investigated. Finally, a linear model is created using the best features and the mean ratings and its prediction efficiency is evaluated and discussed. Copyright: © 2011 Trochidis et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License 3.0 Unported, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Emotion judgments and five channels of physiological data were obtained from 60 participants listening to 60 music excerpts. Various machine learning (ML) methods were used to model the emotion judgments inclusive of neural networks, linear regression, and random forests. Input for models of perceived emotion consisted of audio features extracted from the music recordings. Input for models of felt emotion consisted of physiological features extracted from the physiological recordings. Models were trained and interpreted with consideration of the classic debate in music emotion between cognitivists and emotivists. Our models supported a hybrid position wherein emotion judgments were influenced by a combination of perceived and felt emotions. In comparing the different ML approaches that were used for modeling, we conclude that neural networks were optimal, yielding models that were flexible as well as interpretable. Inspection of a committee machine, encompassing an ensemble of networks, revealed that arousal judgments were predominantly influenced by felt emotion, whereas valence judgments were predominantly influenced by perceived emotion.

Abstract
This project will investigate the link between physical expression and emotions in music.  Since the emotive nature of music is already well known, it will focus on the following questions:
1. Is physical expression a natural result of emotions?
2. Alternatively, can emotions be induced by physical expression?
Section A will explore the answers to these questions using musicology.  After demonstrating that bodily movements both express and induce emotions, it will deal briefly with the possible danger of manipulation.  In section B, the Bible’s teaching on bodily expressions used in worship will be investigated.  After a broader framework of holistic worship has been established, it will explore some key worship terms in Hebrew and Greek.  Examples of physical gestures used in worship in the Bible will be given, followed by physical gestures used in musical worship.  In section C, musicology and the Bible will be brought into dialogue with each other.  In particular, the bi-directional nature of physical expression and emotions in music will be discussed from the perspective of both fields of study.  Practical implications for corporate worship involving music will be teased out .  Finally, it will be established that physical gestures should be used in corporate worship and concluding recommendations will be made.

In this paper, the automated detection of emotion in music is modeled as a multilabel classification task, where a piece of music may belong to more than one class. Four algorithms are evaluated and compared in this task. Furthermore, the predictive power of several audio features is evaluated using a new multilabel feature selection method. Experiments are conducted on a set of 593 songs with 6 clusters of music emotions based on the Tellegen-Watson-Clark model. Results provide interesting insights into the quality of the discussed algorithms and features.

Electroencephalography (EEG)-based emotion classification during music listening has gained increasing attention nowadays due to its promise of potential applications such as musical affective brain-computer interface (ABCI), neuromarketing, music therapy, and implicit multimedia tagging and triggering. However, music is an ecologically valid and complex stimulus that conveys certain emotions to listeners through compositions of musical elements. Using solely EEG signals to distinguish emotions remained challenging. This study aimed to assess the applicability of a multimodal approach by leveraging the EEG dynamics and acoustic characteristics of musical contents for the classification of emotional valence and arousal. To this end, this study adopted machine-learning methods to systematically elucidate the roles of the EEG and music modalities in the emotion modeling. The empirical results suggested that when whole-head EEG signals were available, the inclusion of musical contents did not improve the classification performance. The obtained performance of 74∼76% using solely EEG modality was statistically comparable to that using the multimodality approach. However, if EEG dynamics were only available from a small set of electrodes (likely the case in real-life applications), the music modality would play a complementary role and augment the EEG results from around 61-67% in valence classification and from around 58-67% in arousal classification. The musical timber appeared to replace less-discriminative EEG features and led to improvements in both valence and arousal classification, whereas musical loudness was contributed specifically to the arousal classification. The present study not only provided principles for constructing an EEG-based multimodal approach, but also revealed the fundamental insights into the interplay of the brain activity and musical contents in emotion modeling.

Electroacoustic music with video is now frequently programmed in classical concert venues. There are many different ways in which video and sound can be organized together into an effective work of art. But after attending a number of recent concerts in which pieces for electroacoustic music with video were performed, it struck me that in many of these pieces there is only the most minimal relationship between the video and the audio.  In this article on New Music Box I explore ways in which video and electroacoustic audio can be more successfully combined, based on  techniques used in sound for film.  You can also see the article online at:
http://www.newmusicbox.org/articles/electroacoustic-music-with-video-comparison-with-sound-for-film/

Müzik kavramı bir lisanı, bu lisanın alfabesini ve bu dilin edebiyatını ifade eder. Tıpkı birincil anlamı ile akla gelen edebiyatta olduğu gibi, müzikte de bir olgu farklı biçimler ve farklı edebi kurulumlarla ifade edilebilir. Müziğin duygular ile ilişkisinin gözlenmeye başladığı ilk zamanlardan itibaren, müzik dilinin olgular ve duyguların ifadesi konusundaki zenginliği, bu disiplinin çalışanlarını müzik -duygu ilişkisini teoriye dökme konusunda oldukça cezbetmiştir. Konfiçyus' un Büyük Bilgi 'sinden 21. yüzyılın Psikoakustik çalışmalarına varıncaya dek konu ile ilgili yüzlerce kaynak yazılmış, bazıları bilimsel çalışmalar ile desteklenebilmekle birlikte, çoğunluğu "uygun koşullarda" doğruluk payı sağlayabilmekte yahut herhangi bir kanıta dayandırılamamaktadır.

It has been observed from recent studies that corticolimbic Theta rhythm from EEG
recordings perceived as fear or threatening scene during neural processing of visual
stimuli. In additions, neural oscillations’ patterns in Theta, Alpha and Beta sub-bands
also play important role in brain’s emotional processing. Inspired from these findings,
in this paper we attempt to classify two different emotional states by analyzing
single-channel EEG recordings. A video clip that can evoke 3 different emotional
states: neutral, relaxation and scary is shown to 19 college-aged subjects and they were asked to score their emotional outcome by giving a number between 0 to 10 (where 0 means not scary at all and 10 means the most scary). First, recorded EEG data were preprocessed by stationary wavelet transform (SWT) based artifact removal algorithm. Then power distribution in simultaneous time-frequency domain was analyzed using short-time Fourier transform (STFT) followed by calculating the average power during each 0.2s time-segment for each brain sub-band. Finally, 46 features, as the mean power of frequency bands between 4 and 50 Hz during each time-segment, containing 689 instances—for each subject —were collected for classification. We found that relaxation and fear emotions evoked during watching scary and relaxing movies can be classified with average classification rate of 94.208% using K-NN by applying methods and materials proposed in this paper. We also classified the dataset using SVM and we found out that K-NN classifier (when k = 1) outperforms SVM in classifying EEG dynamics induced by horror and relaxing movies, however, for K > 1 in K-NN, SVM has better average classification rate.

In this paper we propose a neural network model for human emotion and gesture classification. We demonstrate that the proposed architecture represents an effective tool for real-time processing of customer's behavior for distributed on-land systems, such as information kiosks, automated cashiers and ATMs. The proposed approach combines most recent biometric techniques with the neural network approach for real-time emotion and behavioral analysis. In the series of experiments, emotions of human subjects were recorded, recognized, and analyzed to give statistical feedback of the overall emotions of a number of targets within a certain time frame. The result of the study allows automatic tracking of user's behavior based on a limited set of observations. 1. Introduction Recognition of human behavior can be most efficiently achieved by visually detecting facial features and specific body movements, such as gestures. Using computer vision and machine learning algorithms for processing these features, recorded by infrared cameras, we can classify emotional states and behavioral patterns of multiple targets. The aim of this paper is to provide statistical observations and measurements of human behavior during the standard interaction with a user interface of a commonly used software. For academic purposes, we have chosen a very limited number of emotional states and behavioral patterns by studying only type of such standard interaction: the interaction of a user with typical ATM equipment, since it provides us with very distinctive patterns of 'typical' and 'non-typical' behavior and facial expressions. During this study, we observed the behavior of human subjects during standard interaction with the ATM versus non-standard interaction. Automated analysis of these behaviors with the machine learning techniques allowed us to train a complex convolutional neural network (CNN) to classify behavior of a user by classification both body movements and facial features. Such a feedback can provide important measures for user response to an interaction with any chosen system with a limited number of gestures involved. We use infrared cameras to automatically detect features and the movements of the limbs in order to classify user behavior into typical or untypical for the kind of task he is performing. We restrict ourselves to only one type of interaction; however, this kind of classification task is very useful in the number of applications, where the number of gestures of the human is limited, such as: • Customers at the various types of automated machines. For this category of users, the algorithm can be used for detection of unusual/fraudulent behavior to decrease the workload of the closed-circuit television (CCTV), or video surveillance, operators who monitor users of these machines: o customer at the ATM machine; o customer at the ticket machine in the underground; o customer at the automated cashier in the countries, where such payment type is widely used; • Drivers. For this category of users, the algorithm can be used for detection of dangerous actions and preventing the unwanted consequences, such as sleeping, loss of attention etc.: o train driver in the train line/underground; o track driver; o automobile driver;

Speech processing is the study of speech signals, and the methods used to process them. In application such as speech coding, speech synthesis, speech recognition and speaker recognition technology, speech processing is employed. In speech classification, the computation of prosody effects from speech signals plays a major role. In emotional speech signals pitch and frequency is a most important parameters. Normally, the pitch value of sad and happy speech signals has a great difference and the frequency value of happy is higher than sad speech. But, in some cases the frequency of happy speech is nearly similar to sad speech or frequency of sad speech is similar to happy speech. In such situation, it is difficult to recognize the exact speech signal. To reduce such drawbacks, in this paper we propose a Telugu speech emotion classification system with three features like Energy Entropy, Short Time Energy, Zero Crossing Rate and K-NN classifier for the classification. Features are extracted from the speech signals and given to the K-NN. The implementation result shows the effectiveness of proposed speech emotion classification system in classifying the Telugu speech signals based on their prosody effects. The performance of the proposed speech emotion classification system is evaluated by conducting cross validation on the Telugu speech database.

Spectral and excitation features, commonly used in automatic emotion classification systems, parameterise different aspects of the speech signal. This paper groups these features as speech production cues, broad spectral measures and detailed spectral measures and looks at how they differ in their performance in both speaker dependent and speaker independent systems. The extent of speaker normalisation on these features is also considered. Combinations of different features are then compared in terms of classification accuracies. Evaluations were conducted on the LDC emotional speech corpus for a fiveclass problem. Results indicate that MFCCs are very discriminative but suffer from speaker variability. Further, results suggest that the best front end for a speaker independent system is a combination of pitch, energy and formant information.

Many current electroacoustic works are weakened by not having clear structure.  Why is this happening, and why now?  In the following article on New Music Box I consider a number of reasons why this may be occurring.  I begin with considering what the smallest structural unit in music might be, and focus most of the discussion on structural issues involving timbre.  You can see the article online as well: 
http://www.newmusicbox.org/articles/structural-and-playback-issues-in-current-electroacoustic-music/

In recent years, there are many great successes in using deep architectures for unsupervised feature learning from data, especially for images and speech. In this paper, we introduce recent advanced deep learning models to classify two emotional categories (positive and negative) from EEG data. We train a deep belief network (DBN) with differential entropy features extracted from multichannel EEG as input. A hidden markov model (HMM) is integrated to accurately capture a more reliable emotional stage switching. We also compare the performance of the deep models to KNN, SVM and Graph regularized Extreme Learning Machine (GELM). The average accuracies of DBN-HMM, DBN, GELM, SVM, and KNN in our experiments are 87.62%, 86.91%, 85.67%, 84.08%, and 69.66%, respectively. Our experimental results show that the DBN and DBN-HMM models improve the accuracy of EEGbased emotion classification in comparison with the state-ofthe-art methods.

Continuous emotion prediction in the arousal-valence space is now being used in various modalities: music, facial expressions, gestures, text, etc. In order to be able to compare the work of different research groups effectively, we believe it is necessary to set certain guidelines for how to
conduct research—the choice of evaluation metrics of emotion recognition algorithms in particular. In this paper we focus on the field of musical emotion recognition and describe a study designed to discover people’s instinctive preference among the most commonly used evaluation techniques. We gather strong evidence that root mean squared error or Kullback-Leibler divergence should be used for regression based approaches. The raw study data we collected is made publicly available.

Enabling computer systems to recognize facial expressions and infer emotions from them in real time presents a challenging research topic. In this paper, we present a real time approach to emotion recognition through facial expression in live video. We employ an automatic facial feature tracker to perform face localization and feature extraction. The facial feature displacements in the video stream are used as input to a Support Vector Machine classifier. We evaluate our method in terms of recognition accuracy for a variety of interaction and classification scenarios. Our person-dependent and person-independent experiments demonstrate the effectiveness of a support vector machine and feature tracking approach to fully automatic, unobtrusive expression recognition in live video. We conclude by discussing the relevance of our work to affective and intelligent man-machine interfaces and exploring further improvements.

The following paper presents parameterization of emotional speech using perceptual coefficients as well as a comparison of Mel Frequency Cepstral Coefficients (MFCC), Bark Frequency Cepstral Coefficients (BFCC), Perceptual Linear Prediction Coefficients (PLP) and Revised Perceptual Linear Prediction Coefficients (RPLP). Analysis was performed on two different databases: Database of Polish Emotional Speech and the most commonly used for emotion recognition -Berlin Database of Emotional Speech. Both consist of acted emotional speech grouped into six classes of primary emotions. Emotion classification was performed using k-nn algorithm.

Özetçe Literatürde birbirinden farklı sınıflandırma algoritmalarının belirli bir problem için performanslarının karşılaştırılması oldukça yaygın bir uygulama olarak karşımıza çıkmaktadır. Ancak bu çalışmalardan elde edilen sonuçların geçerliliği farklı sınıflandırma algoritmaları karşılaştırırken seçilen model parametrelerine, başarımı ölçmek için kullanılan yöntem ve kriterlere bağlı olacaktır. Dolayısı ile şu sorular cevaplandırılmalıdır: Bir sınıflandırıcının herhangi bir sınıflandırma problemi için yüksek performans gösteriyor olması sözkonusu sınıflandırıcının incelenen sınıflandırma problemi için "en uygun" sınıflandırıcı olduğu iddiasını haklı çıkartır mı? Veya diğer sınıflandırıcıların aynı problem için uygun olmadığı sonucuna bizi ulaştırabilir mi? Sınıflandırıcılar arasında nesnel bir karşılaştırma nasıl yapılabilir? Bu çalışmada. Destek Vektör Makinaları, Yapay Sinir Ağları, Radyal Tabanlı Fonksiyon ve k-En Yakın Komşuluk Algoritmaları ses işaretlerinden duygu sınıflandırma probleminde kullanılarak elde edilen sonuçlardan hareketle yukarıdaki sorular ele alınacaktır

Özetçe Literatürde birbirinden farklı sınıflandırma algoritmalarının belirli bir problem için performanslarının karşılaştırılması oldukça yaygın bir uygulama olarak karşımıza çıkmaktadır. Ancak bu çalışmalardan elde edilen sonuçların geçerliliği farklı sınıflandırma algoritmaları karşılaştırırken seçilen model parametrelerine, başarımı ölçmek için kullanılan yöntem ve kriterlere bağlı olacaktır. Dolayısı ile şu sorular cevaplandırılmalıdır: Bir sınıflandırıcının herhangi bir sınıflandırma problemi için yüksek performans gösteriyor olması sözkonusu sınıflandırıcının incelenen sınıflandırma problemi için "en uygun" sınıflandırıcı olduğu iddiasını haklı çıkartır mı? Veya diğer sınıflandırıcıların aynı problem için uygun olmadığı sonucuna bizi ulaştırabilir mi? Sınıflandırıcılar arasında nesnel bir karşılaştırma nasıl yapılabilir? Bu çalışmada. Destek Vektör Makinaları, Yapay Sinir Ağları, Radyal Tabanlı Fonksiyon ve k-En Yakın Komşuluk Algoritmaları ses işaretlerinden duygu sınıflandırma probleminde kullanılarak elde edilen sonuçlardan hareketle yukarıdaki sorular ele alınacaktır

In this paper, the automated detection of emotion in music is modeled as a multilabel classification task, where a piece of music may belong to more than one class. Four algorithms are evaluated and compared in this task. Furthermore, the predictive power of several audio features is evaluated using a new multilabel feature selection method. Experiments are conducted on a set of 593 songs with 6 clusters of music emotions based on the Tellegen-Watson-Clark model. Results provide interesting insights into the quality of the discussed algorithms and features.

Exploring how people represent natural categories is a key step toward developing a better understanding of how people learn, form memories, and make decisions. Much research on categorization has focused on artificial categories that are created in the laboratory, since studying natural categories defined on high-dimensional stimuli such as images is methodologically challenging. Recent work has produced methods for identifying these representations from observed behavior, such as reverse correlation (RC). We compare RC against an alternative method for inferring the structure of natural categories called Markov chain Monte Carlo with People (MCMCP). Based on an algorithm used in computer science and statistics, MCMCP provides a way to sample from the set of stimuli associated with a natural category. We apply MCMCP and RC to the problem of recovering natural categories that correspond to two kinds of facial affect (happy and sad) from realistic images of faces. Our results show that MCMCP requires fewer trials to obtain a higher quality estimate of people's mental representations of these two categories.

During everyday interaction people display various non-verbal signals that convey emotions. These signals are multi-modal and range from facial expressions, shifts in posture, head pose, and non-verbal speech. They are subtle, continuous and complex. Our work concentrates on the problem of automatic recognition of emotions from such multimodal signals. Most of the previous work has concentrated on classifying emotions as belonging to a set of categories, or by discretising the continuous dimensional space. We propose the use of Continuous Conditional Random Fields (CCRF) in combination with Support Vector Machines for Regression (SVR) for modeling continuous emotion in dimensional space. Our Correlation Aware Continuous Conditional Random Field (CA-CCRF) exploits the non-orthogonality of emotion dimensions. By using visual features based on geometric shape and appearance, and a carefully selected subset of audio features we show that our CCRF and CA-CCRF approaches outperform previously published baselines for all four affective dimensions of valence, arousal, power and expectancy.

In this paper, we propose a neural network model for human emotion and gesture classification. We demonstrate that the proposed architecture represents an effective tool for real-time processing of customer's behavior for distributed on-land systems, such as information kiosks, automated cashiers and ATMs. The proposed approach combines most recent biometric techniques with the neural network approach for real-time emotion and behavioral analysis. In the series of experiments, emotions of human subjects were recorded, recognized, and analyzed to give statistical feedback of the overall emotions of a number of targets within a certain time frame. The result of the study allows automatic tracking of user’s behavior based on a limited set of observations.

Emotion recognition has been a research topic in the field of Human Computer Interaction (HCI) during recent years. Computers have become an inseparable part of human life. Users need human-like interaction to better communicate with computers. Many researchers have become interested in emotion recognition and classification using different sources. A hybrid approach of audio and text has been recently introduced. All such approaches have been done to raise the accuracy and appropriateness of emotion classification. In this study, a hybrid approach of audio and video has been applied for emotion recognition. The innovation of this approach is selecting the characteristics of audio and video and their features as a unique specification for classification. In this research, the SVM method has been used for classifying the data in the SAVEE database. The experimental results show the maximum classification accuracy for audio data is 91.63% while by applying the hybrid approach the accuracy achieved is 99.26%.

Music has grown into an important part of people's daily lives. As we move further into the digital age in which a large collection of music is being created daily and becomes easily accessible renders people to spend more time on activities that involve music. Consequently, the form of music retrieval is changed from catalogue based searches to searches made based on emotion tags in order for easy and effective musical information access. In this study, it is aimed to generate a model for automatic recognition of the perceived emotion of songs with the help of their lyrics and machine learning algorithms. For this purpose, first 300 songs are selected and annotated by human taggers with respect to their perceived emotions. Thereafter, Unigram, Bigram and Trigram word features are extracted from song lyrics after performing text preprocessing where stemming of the Turkish words is an essential part. Then, term by document matrices are created where term frequencies and tf-idf scores are considered as representations for the indices. Five different classification algorithms are fed with these matrices in order to find the best combination that achieves the highest accuracy results where recall and precision values are used as comparison metrics. As a result, best accuracy results are obtained by using Multinomial Naïve Bayes classifier where Unigram features are used to create the term by document matrix. In this setting, Unigram features are stemmed by Zemberek Long stemming method, and the index representation is chosen as term frequency. For this combination, obtained recall and precision values are 43.7 and 46.9, respectively.