Evaluating Deep Learning Models for Music Emotion Recognition

In music information retrieval (MIR), emotion-based classification is a complex and challenging task for researchers. In modern-day information technology, understanding music emotions through humancomputer interaction plays a vital role in capturing the attention of both researchers and the music industry. This paper presents a learning algorithm by adopting the decision tree, random forest, knearest neighbors, multi-layer perceptron (MLP), long short-term memory (LSTM) neural network, convolutional neural networks (CNNs), and CNN-LSTM hybrid deep learning approaches with relevant feature extraction techniques. Many researchers have performed emotion recognition in music for different languages such as English, Chinese, Spanish, Turkish, Hindi, etc. However, languages like Assamese have drawn very little attention in the research of music emotion recognition (MER). This work aims to perform a novel approach to emotion recognition in Assamese songs. In this study, a newly created Assamese dataset of 200 song samples is used with four different emotions and another dataset used is the RAVDESS emotional song database which consists of 1012 song samples with six different music emotions. Relevant features such as mel-frequency cepstrum coefficients (MFCC), mel spectrogram, and chroma features are extracted from the song samples to investigate the performance of the proposed method. A comparative analysis using different classifiers is carried out and the findings of this study suggest that the CNN-LSTM model has shown better accuracy with both datasets. The accuracy is 85.00% with the Assamese dataset, and with the RAVDESS dataset, the accuracy is 89.66% compared to the other classifiers used in this work.

2020

Emotions are affective states which show an individual’s response to mental stimulus. In this paper, we propose the use of Deep Learning and music therapy to influence negative emotions associated with dementia episodes – (Anger, Fear, Sadness and Confusion) and therefore help the patient achieve neutral or positive emotion states – (Joy and Trust). Targeted emotions were identified with the help of a Convoluted Neural Network (CNN) model built on curated datasets containing annotated emotion-labelled audio clips. These audio clips were converted to Mel Frequency Spectrograms (MFS) for classification according to the emotions they represent. We then selected music according to this emotional state using content-based filtering. We achieved average classification results of accuracy: 95.25%, recall: 72.44%, precision: 87.42% and F1 score: 0.7922.

2015

In this paper we describe our approach for the MediaEval's "Emotion in Music" task. Our method consists of deep Long-Short Term Memory Recurrent Neural Networks (LSTM-RNN) for dynamic Arousal and Valence regression, using acoustic and psychoacoustic features extracted from the songs that have been previously proven as effective for emotion prediction in music. Results on the challenge test demonstrate an excellent performance for Arousal estimation (r = 0.613 ± 0.278), but not for Valence (r = 0.026 ± 0.500). Issues regarding the quality of the test set annotations' reliability and distributions are indicated as plausible justifications for these results. By using a subset of the development set that was left out for performance estimation, we could determine that the performance of our approach may be underestimated for Valence (Arousal: r = 0.596 ± 0.386; Valence: r = 0.458 ± 0.551).

National Conference on Artificial Intelligence, 2021

Emotion is a complicated notion present in music that is hard to capture even with fine-tuned feature engineering. In this paper, we investigate the utility of state-of-the-art pre-trained deep audio embedding methods to be used in the Music Emotion Recognition (MER) task. Deep audio embedding methods allow us to efficiently capture the high dimensional features into a compact representation. We implement several multi-class classifiers with deep audio embeddings to predict emotion semantics in music. We investigate the effectiveness of L 3 -Net and VGGish deep audio embedding methods for music emotion inference over four music datasets. The experiments with several classifiers on the task show that the deep audio embedding solutions can improve the performances of the previous baseline MER models. We conclude that deep audio embeddings represent musical emotion semantics for the MER task without expert human engineering.

Sensors

The production and consumption of music in the contemporary era results in big data generation and creates new needs for automated and more effective management of these data. Automated music mood detection constitutes an active task in the field of MIR (Music Information Retrieval). The first approach to correlating music and mood was made in 1990 by Gordon Burner who researched the way that musical emotion affects marketing. In 2016, Lidy and Schiner trained a CNN for the task of genre and mood classification based on audio. In 2018, Delbouys et al. developed a multi-modal Deep Learning system combining CNN and LSTM architectures and concluded that multi-modal approaches overcome single channel models. This work will examine and compare single channel and multi-modal approaches for the task of music mood detection applying Deep Learning architectures. Our first approach tries to utilize the audio signal and the lyrics of a musical track separately, while the second approach applie...

International Journal of Scientific Research in Computer Science, Engineering and Information Technology, 2022

Music Emotion Recognition (MER) is an interesting research topic in artificial intelligence field for recognizing the emotions from the music. The recognition methods and tools for the music signals are growing fast recently. With recent development of the signal processing, machine learning and algorithm optimization, the recognition accuracy is approaching perfection. In this research we are focused on three different significant parts of MER, that are features, learning methods and music emotion theory, to explain and illustrate how to effectively build MER systems. Numerous music players have been created with capabilities like fast forward, backward, variable playback speed (seek and time compression), local playback, and streaming playback with multicast broadcasts in the modern world due to the rapid improvements in multimedia and technology. Although these capabilities serve the user’s fundamental needs, the user is still required to actively browse through the music playlist and choose songs depending on his present state of mind and behavior. Here we are using tensoflow, mediapipe, cv2 library for training data using the face expressions. After training, model would be able to recognize face and by streamlit library from the expressions by the user it will suggest songs playlist and user would be able to play the song by his/her choice.

2023

Most of the existing music recommendation systems use collaborative or content based recommendation engines. However, the music choice of a user is not only dependent to the historical preferences or music contents. But also dependent to the mood of that user. This paper proposes an emotion based music recommendation framework that learns the emotion of a user from the signals obtained via wearable physiological sensors. In particular, the emotion of a user is classified by a wearable computing device which is integrated with a galvanic skin response (GSR) and photo plethysmography (PPG) physiological sensors. This emotion information is feed to any collaborative or content based recommendation engine as a supplementary data. Thus, existing recommendation engine performances can be increased using these data. Therefore, in this paper emotion recognition problem is considered as arousal and valence prediction from multi-channel physiological signals. Experimental results are obtained on 32 subjects' GSR and PPG signal data with/out feature fusion using decision tree, random forest, support vector machine and k-nearest neighbors algorithms. The results of comprehensive experiments on real data confirm the accuracy of the proposed emotion classification system that can be integrated to any recommendation engine.

Electronics, 2021

The paper presents an application for automatically classifying emotions in film music. A model of emotions is proposed, which is also associated with colors. The model created has nine emotional states, to which colors are assigned according to the color theory in film. Subjective tests are carried out to check the correctness of the assumptions behind the adopted emotion model. For that purpose, a statistical analysis of the subjective test results is performed. The application employs a deep convolutional neural network (CNN), which classifies emotions based on 30 s excerpts of music works presented to the CNN input using mel-spectrograms. Examples of classification results of the selected neural networks used to create the system are shown.

2014

Music as a form of art is intentionally composed to be emotionally expressive. The emotional features of music are invaluable for music indexing and recommendation. In this paper we present a cross-comparison of automatic emotional analysis of music. We created a public dataset of Creative Commons licensed songs. Using valence and arousal model, the songs were annotated both in terms of the emotions that were expressed by the whole excerpt and dynamically with 1 Hz temporal resolution. Each song received 10 annotations on Amazon Mechanical Turk and the annotations were averaged to form a ground truth. Four different systems from three teams and the organizers were employed to tackle this problem in an open challenge. We compare their performances and discuss the best practices. While the effect of a larger feature set was not very apparent in the static emotion estimation, the combination of a comprehensive feature set and a recurrent neural network that models temporal dependencies has largely outperformed the other proposed methods for dynamic music emotion estimation.

2021

It is said that health is wealth. Here, health refers to both physical health and mental health. People take various measures to take care of their physical health but ignore their mental health which can lead to depression and even diseases like diabetes mellitus and so on. Emotion detection can help us to diagnose our mental health status. Therefore, this paper proposes a theory for emotion detection and then a recommendation of a song to enhance the user's mood by using the features provided by deep learning and image processing. Here, convolutional neural network-based (CNN) LeNet architecture has been used for emotion detection. The KDEF dataset is used for feeding input to the CNN model and then training it. The model has been trained for detecting the emotion. After training the model, a training accuracy of 98.03% and a validation accuracy of 97.96% have been achieved for correctly recognizing the seven different emotions, that is, sad, disgust, happy, afraid, neutral, angry and surprise through facial expressions.