Academia.eduAcademia.edu

Figure 27 – uploaded by OTHMAN O KHALIFA

See full PDFdownloadDownload figure
Fig. 27. Actor expressing anger (self-collected test data). Fig. 28. Actor expressing anger (RAVESSD database).

Figure 27 Actor expressing anger (self-collected test data). Fig. 28. Actor expressing anger (RAVESSD database).

Related Figures (29)

Fig. 2. Speech emotion recognition system block diagram.
Fig. 2. Speech emotion recognition system block diagram.
Table. 1 The fundamental emotions by Robert Plutchik [13]. Robert Plutchik graphically represented his model on a color wheel known as ’Plutchik’s wheel’ shown in Fig. 1. Plutchik’s wheel shows that there are 8 different fundamental emotions which are joy, trust, fear,
Table. 1 The fundamental emotions by Robert Plutchik [13]. Robert Plutchik graphically represented his model on a color wheel known as ’Plutchik’s wheel’ shown in Fig. 1. Plutchik’s wheel shows that there are 8 different fundamental emotions which are joy, trust, fear,
Fig. 1. Plutchik’s Wheel of Emotions.
Fig. 1. Plutchik’s Wheel of Emotions.
Pre-emphasis: Pre-emphasizing includes filtering the sample word to increase the energy of the signal at higher frequencies. Frame blocking: this step involves segmenting the signal into small durations (frames) ranging between 20ms to 40ms. The speech is signal divided into N samples, the adjacent frames will be set rates by M(M<N). M and N values are 100 and 256 respectively. Fast Fourier Transform: FFT is converting signals from time domain into frequency domain. Mel Filter bank: voices signal in FFT uses triangular band pass Filter. To obtain smooth magnitude spectrum, and reduce the size of the involved features, a set of triangular filters are multiplied by the magnitude frequency response. this step is the final categorized emotion. Various Cc Cc H Cc assi assi fication methods were used to categorize utterance-level features, some example of a famous fication methods are: neural network, hidden Markov models (HMM), and (GMM) Gaussian mixture mode . However conventional classifying methods like MM and GMM may not give a high and accurate assi fying rate compared to deep learning and neural network methods. Moreover hybrid methods [21] may give better results compared to individual methods. Pitch is the most important property of speech. It is a known fact that speech is nothing but a wave that is generated by vibrating objects in a medium such as air. Speech emotion recognition’s researches have always considered the characteristics of pitch in order to have an accurate emotion classification. Many methods were invented to extract features from pitch for example the Cepstral method. In Cepstral method, the signal goes into several stages. First, the analog signal will be sampled and quantized to digitize the signal. The digital signal is then framed in a suitable size and this can be obtained by passing the signal into a hamming window
Pre-emphasis: Pre-emphasizing includes filtering the sample word to increase the energy of the signal at higher frequencies. Frame blocking: this step involves segmenting the signal into small durations (frames) ranging between 20ms to 40ms. The speech is signal divided into N samples, the adjacent frames will be set rates by M(M<N). M and N values are 100 and 256 respectively. Fast Fourier Transform: FFT is converting signals from time domain into frequency domain. Mel Filter bank: voices signal in FFT uses triangular band pass Filter. To obtain smooth magnitude spectrum, and reduce the size of the involved features, a set of triangular filters are multiplied by the magnitude frequency response. this step is the final categorized emotion. Various Cc Cc H Cc assi assi fication methods were used to categorize utterance-level features, some example of a famous fication methods are: neural network, hidden Markov models (HMM), and (GMM) Gaussian mixture mode . However conventional classifying methods like MM and GMM may not give a high and accurate assi fying rate compared to deep learning and neural network methods. Moreover hybrid methods [21] may give better results compared to individual methods. Pitch is the most important property of speech. It is a known fact that speech is nothing but a wave that is generated by vibrating objects in a medium such as air. Speech emotion recognition’s researches have always considered the characteristics of pitch in order to have an accurate emotion classification. Many methods were invented to extract features from pitch for example the Cepstral method. In Cepstral method, the signal goes into several stages. First, the analog signal will be sampled and quantized to digitize the signal. The digital signal is then framed in a suitable size and this can be obtained by passing the signal into a hamming window
Fig. 5. Conventional neural networks [23]. Conventional neural networks are constructed using simple and connected neurons also known as processors, theses neurons produce a sequence of real-value activation, some input neurons get activated through an environmental sensing sensors while other get their activation through assigned weight. Fig. 5 shows a simple neural network with neuron inputs X;,, the weight W,,; and the neuron which sums the weights multiplied by the input.
Fig. 5. Conventional neural networks [23]. Conventional neural networks are constructed using simple and connected neurons also known as processors, theses neurons produce a sequence of real-value activation, some input neurons get activated through an environmental sensing sensors while other get their activation through assigned weight. Fig. 5 shows a simple neural network with neuron inputs X;,, the weight W,,; and the neuron which sums the weights multiplied by the input.
Fig. 7. Illustration of: (a) a Multilayer Perceptron and (b) a Recurrent Neural Network. connection between the layers and the visibility and the hiddenness. It is interesting to note that in DBNs, there is no connection between units if they are from the same layer. The greedy strategy adapts just the highlights of one layer at any given moment and it never straightens
Fig. 7. Illustration of: (a) a Multilayer Perceptron and (b) a Recurrent Neural Network. connection between the layers and the visibility and the hiddenness. It is interesting to note that in DBNs, there is no connection between units if they are from the same layer. The greedy strategy adapts just the highlights of one layer at any given moment and it never straightens
Fig. 8. The connection between the layers.
Fig. 8. The connection between the layers.
Fig. 9. Left: A three-layer Deep Belief Network and a three-layer Deep Boltzmann Machine. Right: Pretraining comprises of learning a heap of adjusted RBM's, that are then formed to make a profound Boltzmann machine [30].
Fig. 9. Left: A three-layer Deep Belief Network and a three-layer Deep Boltzmann Machine. Right: Pretraining comprises of learning a heap of adjusted RBM's, that are then formed to make a profound Boltzmann machine [30].
Fig. 10. Autoencoder of the SDAs model. Stacked Denoising Autoencoders (SDAs) is one of DL architectures that was first introduced in 2008 by Vincet et al [33],[34]. SDAs is an extension of a classical autoencoder as shown in Fig. 10. What makes SDAs a “deep” architecture is that it is constructed by stacking multiple autoencoders together. The unsupervised pre training of each autoencoder is performed in a greedy layer by layer method. Once a SDAs is learnt, its yield will be utilized as the input portrayal of a regulated learning calculation for recognition tasks. So, the main idea is that the created or reconstructed input will have as minimum as possible of errors, so the input value closes to the real value. Notedly, in this method the output, which is in fact corrupted, comes out clean. The idea of an autoencoder can be quickly portrayed here: Given a set of data points x = {x,, X, ....,Xm}, map x to another set of data points Y = {Y%, ,{Y,...,{%m} where n < m. From the compacted set y, reproducing a lot of ~x, which approximates the first information x. The mapping x —y is called “encoding" and the mapping y > ~x is called “decoding".
Fig. 10. Autoencoder of the SDAs model. Stacked Denoising Autoencoders (SDAs) is one of DL architectures that was first introduced in 2008 by Vincet et al [33],[34]. SDAs is an extension of a classical autoencoder as shown in Fig. 10. What makes SDAs a “deep” architecture is that it is constructed by stacking multiple autoencoders together. The unsupervised pre training of each autoencoder is performed in a greedy layer by layer method. Once a SDAs is learnt, its yield will be utilized as the input portrayal of a regulated learning calculation for recognition tasks. So, the main idea is that the created or reconstructed input will have as minimum as possible of errors, so the input value closes to the real value. Notedly, in this method the output, which is in fact corrupted, comes out clean. The idea of an autoencoder can be quickly portrayed here: Given a set of data points x = {x,, X, ....,Xm}, map x to another set of data points Y = {Y%, ,{Y,...,{%m} where n < m. From the compacted set y, reproducing a lot of ~x, which approximates the first information x. The mapping x —y is called “encoding" and the mapping y > ~x is called “decoding".
6. PROPOSED METHODOLOGY type of compression. Authors in [7] used Convolution recurrent neural network stacked on top of 2-layers of LTSM long short-term memory and concluded that their model has clearly outperformed other models in arousal and valence. Other studies preferred to use RNNs and were able to bypass what other studies could not, which is making the model be able to ignore the silence frames of the waveform by assigning very small weight to them which causes them to be ignored in the pooling operation [10]. In [35], a convolutional neural network is proposed as a deep learning model, his design consisted one convolutional layer followed by max pooling layer, his method showed a relatively higher performance than LDA yet a lower performance than other recognition methods namely RDA, KDA, SVM, KNN and CNN [21]. The two methods were later combined. The first used method was GMM-HMM were the system inputs are the MFCC features of the audio signal. The second method was the conventional SVM classifier with LLD (Low Lever Descriptors) as the input of the model. The
6. PROPOSED METHODOLOGY type of compression. Authors in [7] used Convolution recurrent neural network stacked on top of 2-layers of LTSM long short-term memory and concluded that their model has clearly outperformed other models in arousal and valence. Other studies preferred to use RNNs and were able to bypass what other studies could not, which is making the model be able to ignore the silence frames of the waveform by assigning very small weight to them which causes them to be ignored in the pooling operation [10]. In [35], a convolutional neural network is proposed as a deep learning model, his design consisted one convolutional layer followed by max pooling layer, his method showed a relatively higher performance than LDA yet a lower performance than other recognition methods namely RDA, KDA, SVM, KNN and CNN [21]. The two methods were later combined. The first used method was GMM-HMM were the system inputs are the MFCC features of the audio signal. The second method was the conventional SVM classifier with LLD (Low Lever Descriptors) as the input of the model. The
Fig. 11. The proposed architecture of the system (training stage). order to be resampled. The resultant frequency rate should be 16Khz before any processing is implemented. The utterances will then be converted into their respected spectrogram. the spectrogram consists of two axis time and frequency axis, where time is shown as a horizontal axis and the frequency is shown as the vertical axis. The spectrogram is simply a picture that shows the different variation of energy with different frequency at different times. The intensity of the energy will be represented by either a darkness or different colors. The fact that every period of vocal vibration is associated with glottal pulse gives a special importance to the vocal fold vibration. Furthermore, in the preprocessing stage of the proposed model, all the audio files will be converted in a wide-band spectrogram. The hamming window will be fixed along with the overlap. DFT will also be fixed. e.g. (hamming windows is 4ms with 4ms overlap, and 512 DFT points). Notice that any frequency that is greater than 4000hz will be discarded because 4kHz is cindered to be sufficient for speech emotion recognition according to many studies. Taking into consideration that different databases might be tested on this project, it is important that all the audio files go through an antialiasing low pass filter in
Fig. 11. The proposed architecture of the system (training stage). order to be resampled. The resultant frequency rate should be 16Khz before any processing is implemented. The utterances will then be converted into their respected spectrogram. the spectrogram consists of two axis time and frequency axis, where time is shown as a horizontal axis and the frequency is shown as the vertical axis. The spectrogram is simply a picture that shows the different variation of energy with different frequency at different times. The intensity of the energy will be represented by either a darkness or different colors. The fact that every period of vocal vibration is associated with glottal pulse gives a special importance to the vocal fold vibration. Furthermore, in the preprocessing stage of the proposed model, all the audio files will be converted in a wide-band spectrogram. The hamming window will be fixed along with the overlap. DFT will also be fixed. e.g. (hamming windows is 4ms with 4ms overlap, and 512 DFT points). Notice that any frequency that is greater than 4000hz will be discarded because 4kHz is cindered to be sufficient for speech emotion recognition according to many studies. Taking into consideration that different databases might be tested on this project, it is important that all the audio files go through an antialiasing low pass filter in
Fig. 13. (a) Learning curve and overfitting (b) Results of initial model in detecting 10 targeted classes. Majlesi Journal of Electrical Engineering
Fig. 13. (a) Learning curve and overfitting (b) Results of initial model in detecting 10 targeted classes. Majlesi Journal of Electrical Engineering
Fig. 14. Learning curve after splitting the data into training, validation and testing. Another problem with this model is that the testing set scored a higher accuracy than the validation dataset, this indicated that there is a data leakage problem in which the data used in the validation dataset are the same as the one used in the testing dataset. To validate this theory of a data leakage problem, the RAVDESS dataset was divided into 3 different sections: training, validation, and testing, to ensure that data are not used
Fig. 14. Learning curve after splitting the data into training, validation and testing. Another problem with this model is that the testing set scored a higher accuracy than the validation dataset, this indicated that there is a data leakage problem in which the data used in the validation dataset are the same as the one used in the testing dataset. To validate this theory of a data leakage problem, the RAVDESS dataset was divided into 3 different sections: training, validation, and testing, to ensure that data are not used
Fig. 15. (a) Learning curve and overfitting (b) Results of initial model in detecting lasses of female emotions. After splitting the dataset, the number of targeted classes reduced to 5 and, then the data fed into the model in two different times and the results came as follows:
Fig. 15. (a) Learning curve and overfitting (b) Results of initial model in detecting lasses of female emotions. After splitting the dataset, the number of targeted classes reduced to 5 and, then the data fed into the model in two different times and the results came as follows:
Fig. 16. (a) Learning curve and overfitting (b) Results of initial model in detecting classes of male data.
Fig. 16. (a) Learning curve and overfitting (b) Results of initial model in detecting classes of male data.
Fig. 17. (a) Learning curve and overfitting (b) Results of improved model in detecting two emotional classes (Male samples). Majlesi Journal of Electrical Engineering
Fig. 17. (a) Learning curve and overfitting (b) Results of improved model in detecting two emotional classes (Male samples). Majlesi Journal of Electrical Engineering
Fig. 18. (a) Learning curve and overfitting (b) Results of improved model in detecting three emotional classes (Male samples).
Fig. 18. (a) Learning curve and overfitting (b) Results of improved model in detecting three emotional classes (Male samples).
Fig. 19. (a) Learning curve and overfitting (b) Results of improved model in detecting five emotional classes (Male samples).
Fig. 19. (a) Learning curve and overfitting (b) Results of improved model in detecting five emotional classes (Male samples).
Fig. 23. Two classes emotional classifier with (Noise Adding +Time Shifting). "1 This step took place to make sure that the model will not have an underfitting problem. Initially we implemented the augmentation method on the model to classify 2 targeted classes, the goal was to determine the best data augmentation method. The next step was implementing the chosen augmentation method on the model to classify 3 and 5 targeted classes.
Fig. 23. Two classes emotional classifier with (Noise Adding +Time Shifting). "1 This step took place to make sure that the model will not have an underfitting problem. Initially we implemented the augmentation method on the model to classify 2 targeted classes, the goal was to determine the best data augmentation method. The next step was implementing the chosen augmentation method on the model to classify 3 and 5 targeted classes.
The result in the figure above shows a dramatic increase in the validation accuracy. Nevertheless, it is clear from the results obtained that the training valid loss shows a relatively high loss value which is a clear sign of an underfitting problem in the model. We figured that this problem occurred because the model was not fed enough data. The reason that we did not face this problem in the first experiment is that we did not split the dataset into female and male set, the splitting process caused the number of training data to drop significantly.
The result in the figure above shows a dramatic increase in the validation accuracy. Nevertheless, it is clear from the results obtained that the training valid loss shows a relatively high loss value which is a clear sign of an underfitting problem in the model. We figured that this problem occurred because the model was not fed enough data. The reason that we did not face this problem in the first experiment is that we did not split the dataset into female and male set, the splitting process caused the number of training data to drop significantly.
The two methods used were Noise Adding combined with Shifting, Noise Adding combined with Pitch Tuning. The results came as follow:
The two methods used were Noise Adding combined with Shifting, Noise Adding combined with Pitch Tuning. The results came as follow:
Fig. 21. Before data augmentation.
Fig. 21. Before data augmentation.
Fig. 24. Two classes emotional classifier with (Pitch Shifting + Noise Adding).
Fig. 24. Two classes emotional classifier with (Pitch Shifting + Noise Adding).
Fig. 25. Three classes emotional classifier with (Noise Adding + Time Shifting). As we can see from the results above, Noise Adding + Time Shifting method produced the best accuracy, hence I carried the method forward and tested on the two remaining groups (3 classes emotions, 5 classes emotions) and the results came as follow:
Fig. 25. Three classes emotional classifier with (Noise Adding + Time Shifting). As we can see from the results above, Noise Adding + Time Shifting method produced the best accuracy, hence I carried the method forward and tested on the two remaining groups (3 classes emotions, 5 classes emotions) and the results came as follow:
Fig. 26. Five classes emotional classifier with (Noise Adding + time Shifting).
Fig. 26. Five classes emotional classifier with (Noise Adding + time Shifting).
Table 3. Results from life experiment.
Table 3. Results from life experiment.
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved