Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Related Work
Data Collection
From Textual to Visual Sentiment Analysis
Experimental Evaluation
Dataset Preparation
Experimental Settings
Image Analysis Settings
Results
Conclusions
References
All Topics
Computer Science

Cross-Media Learning for Image Sentiment Analysis in the Wild

Profile image of Fabrizio FalchiFabrizio Falchi

2017 IEEE International Conference on Computer Vision Workshops (ICCVW)

https://doi.org/10.1109/ICCVW.2017.45
visibility

…

description

11 pages

link

1 file

Abstract

Much progress has been made in the field of sentiment analysis in the past years. Researchers relied on textual data for this task, while only recently they have started investigating approaches to predict sentiments from multimedia content. With the increasing amount of data shared on social media, there is also a rapidly growing interest in approaches that work "in the wild", i.e. that are able to deal with uncontrolled conditions. In this work, we faced the challenge of training a visual sentiment classifier starting from a large set of user-generated and unlabeled contents. In particular, we collected more than 3 million tweets containing both text and images, and we leveraged on the sentiment polarity of the textual contents to train a visual sentiment classifier. To the best of our knowledge, this is the first time that a cross-media learning approach is proposed and tested in this context. We assessed the validity of our model by conducting comparative studies and evaluations on a benchmark for visual sentiment analysis. Our empirical study shows that although the text associated to each image is often noisy and weakly correlated with the image content, it can be profitably exploited to train a deep Convolutional Neural Network that effectively predicts the sentiment polarity of previously unseen images.

Related papers

Visual Sentiment Analysis Using Deep Learning Models with Social Media Data
Monica Maria Ciobanu

Applied Sciences, 2022

Analyzing the sentiments of people from social media content through text, speech, and images is becoming vital in a variety of applications. Many existing research studies on sentiment analysis rely on textual data, and similar to the sharing of text, users of social media share more photographs and videos. Compared to text, images are said to exhibit the sentiments in a much better way. So, there is an urge to build a sentiment analysis model based on images from social media. In our work, we employed different transfer learning models, including the VGG-19, ResNet50V2, and DenseNet-121 models, to perform sentiment analysis based on images. They were fine-tuned by freezing and unfreezing some of the layers, and their performance was boosted by applying regularization techniques. We used the Twitter-based images available in the Crowdflower dataset, which contains URLs of images with their sentiment polarities. Our work also presents a comparative analysis of these pre-trained mode...

View PDFchevron_right
Diving Deep into Sentiment
Victor Campos

Proceedings of the 1st International Workshop on Affect & Sentiment in Multimedia, 2015

Visual media are powerful means of expressing emotions and sentiments. The constant generation of new content in social networks highlights the need of automated visual sentiment analysis tools. While Convolutional Neural Networks (CNNs) have established a new state-of-the-art in several vision problems, their application to the task of sentiment analysis is mostly unexplored and there are few studies regarding how to design CNNs for this purpose. In this work, we study the suitability of fine-tuning a CNN for visual sentiment prediction as well as explore performance boosting techniques within this deep learning setting. Finally, we provide a deep-dive analysis into a benchmark, state-of-theart network architecture to gain insight about how to design patterns for CNNs on the task of visual sentiment prediction.

View PDFchevron_right
From Image to Text in Sentiment Analysis via Regression and Deep Learning
Liviu P. Dinu

Proceedings - Natural Language Processing in a Deep Learning World

Images and text represent types of content which are used together for conveying user emotions in online social networks. These contents are usually associated with a sentiment category. In this paper, we investigate an approach for mapping images to text for three types of sentiment categories: positive, neutral and negative. The mapping from images to text is performed using a Kernel Ridge Regression model. We considered two types of image features: i) RGB pixel-values features, and ii) features extracted with a deep learning approach. The experimental evaluation was performed on a Twitter data set containing both text and images and the sentiment associated with these. The experimental results show a difference in performance for different sentiment categories, in particular the mapping that we propose performs better for the positive sentiment category in comparison with the neutral and negative ones. Furthermore, the experimental results show that the more complex deep learning features perform better than the RGB pixel-value features for all sentiment categories and for larger training sets.

View PDFchevron_right
"Sentiment Analysis In The Digital Era: Exploring Text, Audio, And Video Modalities With Emphasis On Deep Learning And Future Directions"
REETU AWASTHI

In the contemporary digital landscape, people routinely share their opinions online, prompting extensive research in the realm of Sentiment Analysis-a study focused on understanding user sentiments and feelings towards various entities. This paper defines sentiment and delves into the intricacies of sentiment analysis applied to text, audio, and video data. The challenges inherent in each modality are explored, shedding light on the predominant use of deep learning algorithms, particularly the Convolution Neural Network (CNN), in sentiment analysis tasks. Notably, the study reveals a notable disproportion in research emphasis, with textual data garnering significant attention compared to audio and video data. The research underscores the prevalence of CNN in visual sentiment analysis and advocates for further exploration into sentiment analysis on voice data as a promising area for future research.

View PDFchevron_right
Transformer-Based Feature Fusion Approach for Multimodal Visual Sentiment Recognition Using Tweets in the Wild
Fatimah Alzamzami

IEEE Access

We present an image-based real-time sentiment analysis system that can be used to recognize in-the-wild sentiment expressions on online social networks. The system deploys the newly proposed transformer architecture on online social networks (OSN) big data to extract emotion and sentiment features using three types of images: images containing faces, images containing text, and images containing no faces/text. We build three separate models, one for each type of image, and then fuse all the models to learn the online sentiment behavior. Our proposed methodology combines a supervised two-stage training approach and threshold-moving method, which is crucial for the data imbalance found in OSN data. The training is carried out on existing popular datasets (i.e., for the three models) and our newly proposed dataset, the Domain Free Multimedia Sentiment Dataset (DFMSD). Our results show that inducing the threshold-moving method during the training has enhanced the sentiment learning performance by 5-8% more points compared to when the training was conducted without the threshold-moving approach. Combining the two-stage strategy with the threshold-moving method during the training process, has been proven effective to further improve the learning performance (i.e. by ≈ 12% more enhanced accuracy compared to the threshold-moving strategy alone). Furthermore, the proposed approach has shown a positive learning impact on the fusion of the three models in terms of the accuracy and F-score.

View PDFchevron_right
A Comprehensive Review of Visual-Textual Sentiment Analysis from Social Media Networks
mohammad asadpour

arXiv (Cornell University), 2022

Social media networks have become a significant aspect of people's lives, serving as a platform for their ideas, opinions and emotions. Consequently, automated sentiment analysis (SA) is critical for recognising people's feelings in ways that other information sources cannot. The analysis of these feelings revealed various applications, including brand evaluations, YouTube film reviews and healthcare applications. As social media continues to develop, people publish vast quantities of information in various formats, like text, pictures, audio, and video. Thus, traditional SA algorithms have become limited, as they do not consider the expressiveness of other modalities. By including such characteristics from various material sources, these multimodal data streams provide new opportunities for optimising the expected results beyond text-based SA. Our study focuses on the forefront field of multimodal SA, which examines visual and textual data posted on social media networks. Many people are more likely to utilise this information to express themselves on these platforms. To serve as a resource for academics in this rapidly growing field, we introduce a comprehensive overview of textual and visual SA, including data pre-processing, feature extraction techniques, sentiment benchmark datasets, and the efficacy of multiple classification methodologies suited to each field. We also provide a brief introduction of the most frequently utilised data fusion strategies and a summary of existing research on visual-textual SA. Finally, we highlight the most significant challenges and investigate several important sentiment applications.

View PDFchevron_right
Sentiment Classification using Images and Label Embeddings
Lakshay Sharma

ArXiv, 2017

In this project we analysed how much semantic information images carry, and how much value image data can add to sentiment analysis of the text associated with the images. To better understand the contribution from images, we compared models which only made use of image data, models which only made use of text data, and models which combined both data types. We also analysed if this approach could help sentiment classifiers generalize to unknown sentiments.

View PDFchevron_right
An Overview on Image Sentiment Analysis: Methods, Datasets and Current Challenges
Alessandro Ortis

Proceedings of the 16th International Joint Conference on e-Business and Telecommunications

This paper introduces the research field of Image Sentiment Analysis, analyses the related problems, provides an in-depth overview of current research progress, discusses the major issues and outlines the new opportunities and challenges in this area. An overview of the most significant works is presented. A discussion about the related specific issues is provided: emotions representation models, existing datasets and most used features. A generalizable analysis of the problem is also presented, by identifying and analyzing the components that affect the sentiment toward an image. Furthermore, the paper introduces some additional challenges and techniques that could be investigated, proposing suggestions for new methods, features and datasets.

View PDFchevron_right
Exploiting objective text description of images for visual sentiment analysis
Sebastiano Battiato

Multimedia Tools and Applications

This paper addresses the problem of Visual Sentiment Analysis focusing on the estimation of the polarity of the sentiment evoked by an image. Starting from an embedding approach which exploits both visual and textual features, we attempt to boost the contribution of each input view. We propose to extract and employ an Objective Text description of images rather than the classic Subjective Text provided by the users (i.e., title, tags and image description) which is extensively exploited in the state of the art to infer the sentiment associated to social images. Objective Text is obtained from the visual content of the images through recent deep learning architectures which are used to classify object, scene and to perform image captioning. Objective Text features are then combined with visual features in an embedding space obtained with Canonical Correlation Analysis. The sentiment polarity is then inferred by a supervised Support Vector Machine. During the evaluation, we compared an extensive number of text and visual features combinations and baselines obtained by considering the state of the art methods. Experiments performed on a representative dataset of 47235 labelled samples demonstrate that the exploitation of Objective Text helps to outperform state-of-the-art for sentiment polarity estimation.

View PDFchevron_right
Visual Sentiment Analysis using Deep Learning Models: A Comparative Study
Yasmin Shaikh

Zenodo (CERN European Organization for Nuclear Research), 2023

With the rapid development of social networking websites (SNWs), it has become a well established platform for communication and information exchange with the help of high speed Internet facilities have made it convenient for users to create and maintain their social profile account on online platform through which they can communicate with other users on the website. Analyzing the sentiments of social media user's content through audio, text, image, and video plays an important role in many applications like human behaviour prediction, medical investigation, education mining, and market prediction, recommend system, fraud detection and many such applications. Sentiment analysis is used to predict the emotion or sentiment of user about their shared data. In current environment majority of people share images with substituting text as a common practice to show their feelings on the social networking websites like Facebook, Instagram, twitter as well as on any other platform. Analysis of Multimodal (image, text) data has been paid less attention as compared to single modality (image or text) data. Analysis of visual content is strenuous task in terms of detection and recognition of image data. For identification of emotions from visual content, image

View PDFchevron_right

References (53)

  1. Groundtruth: NEG, Prediction: NEG Groundtruth: NEG, Prediction: NEU Groundtruth: NEG, Prediction: POS Groundtruth: NEU, Prediction: NEG Groundtruth: NEU, Prediction: NEU Groundtruth: NEU, Prediction: POS Groundtruth: POS, Prediction: NEG Groundtruth: POS, Prediction: NEU Groundtruth: POS, Prediction: POS References
  2. G. Amato, F. Carrara, F. Falchi, C. Gennaro, C. Meghini, and C. Vairo. Deep learning for decentralized parking lot occupancy detection. Expert Syst. Appl., 72:327 -334, 2017. 4
  3. G. Amato, F. Falchi, and L. Vadicamo. Visual recognition of ancient inscriptions Using Convolutional Neural Network and Fisher Vector. JOCCH, 9(4):21:1-21:24, Dec. 2016. 4
  4. M. Avvenuti, S. Cresci, F. Del Vigna, and M. Tesconi. Im- promptu crisis mapping to prioritize emergency response. Computer, 49(5):28-37, 2016. 1
  5. S. Baccianella, A. Esuli, and F. Sebastiani. Sentiwordnet 3.0: An enhanced lexical resource for sentiment analysis and opinion mining. In LREC 2010. 5
  6. C. Baecchi, T. Uricchio, M. Bertini, and A. Del Bimbo. A multimodal feature learning approach for sentiment analysis of social network multimedia. Multimedia Tools and Appli- cations, 75(5):2507-2525, 2016. 3
  7. F. Barbieri, V. Basile, D. Croce, M. Nissim, N. Novielli, and V. Patti. Overview of the evalita 2016 sentiment polarity classification task. In EVALITA 2016. 3
  8. A. Bermingham and A. F. Smeaton. Classifying sentiment in microblogs: is brevity an advantage? In CIKM 2010. ACM. 2
  9. D. Borth, R. Ji, T. Chen, T. Breuel, and S.-F. Chang. Large- scale visual sentiment ontology and detectors using adjective noun pairs. In Multimedia 2013. ACM. 1, 2, 8
  10. V. Campos, B. Jou, and X. Giró i Nieto. From pixels to sen- timent: Fine-tuning CNNs for visual sentiment prediction. Image and Vision Computing, 2017. 3, 6, 8
  11. D. Cao, R. Ji, D. Lin, and S. Li. A cross-media public sen- timent analysis system for microblog. Multimedia Systems, 22(4):479-486, 2016. 1, 3
  12. T. Chen, D. Borth, T. Darrell, and S. Chang. Deepsentibank: Visual sentiment concept classification with deep convolu- tional neural networks. CoRR, abs/1410.8586, 2014. 3, 8
  13. F. Chollet. Keras. https://github.com/fchollet/ keras, 2015. 5
  14. A. Cimino and F. Dell'Orletta. Tandem LSTM-SVM ap- proach for sentiment analysis. In EVALITA 2016. 3, 5
  15. S. Cresci, M. Tesconi, A. Cimino, and F. Dell'Orletta. A linguistically-driven approach to cross-event damage assess- ment of natural disasters from social media messages. In WWW Companion 2015. ACM. 1
  16. R. Datta, D. Joshi, J. Li, and J. Z. Wang. Image retrieval: Ideas, influences, and trends of the new age. ACM Comput. Surv., 40(2):5:1-5:60, May 2008. 1
  17. J. Deriu, M. Gonzenbach, F. Uzdilli, A. Lucchi, V. D. Luca, and M. Jaggi. Swisscheese at semeval-2016 task 4: Sentiment classification using an ensemble of convolutional neural networks with distant supervision. In SemEval @ NAACL-HLT 2016. 2
  18. Y. Gal and Z. Ghahramani. A theoretically grounded applica- tion of dropout in recurrent neural networks. In NIPS 2016. 5
  19. A. Go, R. Bhayani, and L. Huang. Twitter sentiment classi- fication using distant supervision. CS224N Project Report, Stanford, 1(12), 2009. 2
  20. M. Hu and B. Liu. Mining and summarizing customer re- views. In SIGKDD 2004. ACM. 5
  21. J. Islam and Y. Zhang. Visual sentiment analysis for so- cial images using transfer learning approach. In BDCloud- SocialCom-SustainCom 2016. IEEE. 3, 6, 8
  22. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Gir- shick, S. Guadarrama, and T. Darrell. Caffe: Convolu- tional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014. 6
  23. B. Jou, T. Chen, N. Pappas, M. Redi, M. Topkara, and S.- F. Chang. Visual affect around the world: A large-scale multilingual visual sentiment ontology. In Multimedia 2015. ACM. 1, 2, 3, 8
  24. A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS 2012. 3, 4, 8
  25. M. Laver, K. Benoit, and J. Garry. Extracting policy po- sitions from political texts using words as data. American Political Science Review, 97(2):311-331, 005 2003. 1
  26. S. Li, Z. Wang, G. Zhou, and S. Y. M. Lee. Semi-supervised learning for imbalanced sentiment classification. In IJCAI 2011. 4
  27. X. Li, T. Uricchio, L. Ballan, M. Bertini, C. G. M. Snoek, and A. D. Bimbo. Socializing the semantic gap: A comparative survey on image tag assignment, refinement, and retrieval. ACM Comput. Surv., 49(1):14:1-14:39, June 2016. 1
  28. Z. Li, Y. Fan, W. Liu, and F. Wang. Image sentiment predic- tion based on textual descriptions with adjective noun pairs. Multimedia Tools and Applications, pages 1-18, 2017. 3, 6
  29. L. Ma, Z. Lu, L. Shang, and H. Li. Multimodal convolutional neural networks for matching image and sentence. In ICCV 2015, pages 2623-2631. IEEE, 2015. 3
  30. J. Machajdik and A. Hanbury. Affective image classifica- tion using features inspired by psychology and art theory. In Multimedia 2010. ACM. 2
  31. J. Mao, W. Xu, Y. Yang, J. Wang, Z. Huang, and A. Yuille. Deep captioning with multimodal recurrent neural networks (m-rnn). arXiv preprint arXiv:1412.6632, 2014. 3
  32. T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. In NIPS 2013. 2, 5
  33. G. Mishne, N. S. Glance, et al. Predicting movie sales from blogger sentiment. In Computational Approaches to Analyz- ing Weblogs 2006. AAAI. 1
  34. S. M. Mohammad, S. Kiritchenko, and X. Zhu. Nrc-canada: Building the state-of-the-art in sentiment analysis of tweets. In SemEval @ NAACL-HLT 2013. 4
  35. P. Nakov, A. Ritter, S. Rosenthal, F. Sebastiani, and V. Stoy- anov. Semeval-2016 task 4: Sentiment analysis in twitter. In SemEval @ NAACL-HLT 2016. 2, 4
  36. B. O'Connor, R. Balasubramanyan, B. R. Routledge, and N. A. Smith. From tweets to polls: Linking text sentiment to public opinion time series. In ICWSM 2010. AAAI. 1
  37. J. Pennington, R. Socher, and C. D. Manning. Glove: Global vectors for word representation. In EMNLP 2014. 2, 5
  38. T. Rao, M. Xu, and D. Xu. Learning multi-level deep rep- resentations for image emotion classification. arXiv preprint arXiv:1611.07145, 2016. 3
  39. A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carls- son. CNN features off-the-shelf: an astounding baseline for recognition. In CVPRW 2014, pages 512-519. IEEE, 2014. 4
  40. M. Rouvier and B. Favre. SENSEI-LIF at semeval-2016 task 4: Polarity embedding fusion for robust sentiment analysis. In SemEval @ NAACL-HLT 2016. 2
  41. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211-252, 2015. 5, 8
  42. M. Schuster and K. K. Paliwal. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11):2673-2681, 1997. 4
  43. S. Siersdorfer, E. Minack, F. Deng, and J. Hare. Analyzing and predicting sentiment of images on the social web. In Multimedia 2010. ACM. 1, 2, 8
  44. K. Simonyan and A. Zisserman. Very deep convolu- tional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014. 4, 5, 8
  45. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In CVPR 2015. IEEE. 3, 8
  46. D. Tang, B. Qin, and T. Liu. Document modeling with gated recurrent neural network for sentiment classification. In EMNLP 2015. 4
  47. T. Tieleman and G. Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2), 2012. 5
  48. Y. Wang, S. Wang, J. Tang, H. Liu, and B. Li. Unsupervised sentiment analysis for social media images. In IJCAI 2015. 3
  49. T. Wilson, J. Wiebe, and P. Hoffmann. Recognizing con- textual polarity in phrase-level sentiment analysis. In HLT- EMNLP 2005. 5
  50. Q. You, J. Luo, H. Jin, and J. Yang. Cross-modality consis- tent regression for joint visual-textual sentiment analysis of social multimedia. In WSDM 2016. ACM. 1, 3
  51. Q. You, J. Luo, H. Jin, and J. Yang. Robust image sentiment analysis using progressively trained and domain transferred deep networks. CoRR, abs/1509.06041, 2015. 1, 3, 6, 8
  52. J. Yuan, S. Mcdonough, Q. You, and J. Luo. Sentribute: Image sentiment analysis from a mid-level perspective. In WISDOM @ SIGKDD 2013. ACM. 2, 8
  53. B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. Learning deep features for scene recognition using places database. In NIPS 2014. 3, 4, 5, 8

Related papers

Visual sentiment prediction with deep convolutional neural networks
roudatul jannah

Images have become one of the most popular types of media through which users convey their emotions within online social networks. Although vast amount of research is devoted to sentiment analysis of textual data, there has been very limited work that focuses on analyzing sentiment of image data. In this work, we propose a novel visual sentiment prediction framework that performs image understanding with Convolutional Neural Networks (CNN). Specifically, the proposed sentiment prediction framework performs transfer learning from a CNN with millions of parameters, which is pre-trained on large-scale data for object recognition. Experiments conducted on two real-world datasets from Twitter and Tumblr demonstrate the effectiveness of the proposed visual sentiment analysis framework.

View PDFchevron_right
From pixels to sentiment: Fine-tuning CNNs for visual sentiment prediction
Victor Campos

Image and Vision Computing, 2017

Visual multimedia have become an inseparable part of our digital social lives, and they often capture moments tied with deep affections. Automated visual sentiment analysis tools can provide a means of extracting the rich feelings and latent dispositions embedded in these media. In this work, we explore how Convolutional Neural Networks (CNNs), a now de facto computational machine learning tool particularly in the area of Computer Vision, can be specifically applied to the task of visual sentiment prediction. We accomplish this through fine-tuning experiments using a state-of-the-art CNN and via rigorous architecture analysis, we present several modifications that lead to accuracy improvements over prior art on a dataset of images from a popular social media platform. We additionally present visualizations of local patterns that the network learned to associate with image sentiment for insight into how visual positivity (or negativity) is perceived by the model.

View PDFchevron_right
Robust Image Sentiment Analysis using Progressively Trained and Domain Transferred Deep Networks
Jiebo Luo

Sentiment analysis of online user generated content is important for many social media analytics tasks. Researchers have largely relied on textual sentiment analysis to develop systems to predict political elections, measure economic indicators, and so on. Recently, social media users are increasingly using images and videos to express their opinions and share their experiences. Sentiment analysis of such large scale visual content can help better extract user sentiments toward events or topics, such as those in image tweets, so that prediction of sentiment from visual content is complementary to textual sentiment analysis. Motivated by the needs in leveraging large scale yet noisy training data to solve the extremely challenging problem of image sentiment analysis, we employ Convolutional Neural Networks (CNN). We first design a suitable CNN architecture for image sentiment analysis. We obtain half a million training samples by using a baseline sentiment algorithm to label Flickr images. To make use of such noisy machine labeled data, we employ a progressive strategy to fine-tune the deep network. Furthermore, we improve the performance on Twitter images by inducing domain transfer with a small number of manually labeled Twitter images. We have conducted extensive experiments on manually labeled Twitter images. The results show that the proposed CNN can achieve better performance in image sentiment analysis than competing algorithms.

View PDFchevron_right
Visual Sentiment Analysis for Social Images Using Transfer Learning Approach
Jyoti Islam

Visual sentiment analysis framework can predict the sentiment of an image by analyzing the image contents. Nowadays, people are uploading millions of images in social networks such as Twitter, Facebook, Google Plus, and Flickr. These images play a crucial part in expressing emotions of users in online social networks. As a result, image sentiment analysis has become important in the area of online multimedia big data research. Several research works are focusing on analyzing the sentiment of the textual contents. However, little investigation has been done to develop models that can predict sentiment of visual content. In this paper, we propose a novel visual sentiment analysis framework using transfer learning approach to predict sentiment. We use hyper-parameters learned from a very deep convolutional neural network to initialize our network model to prevent overfitting. We conduct extensive experiments on a Twitter image dataset and prove that our model achieves better performance than the current state-of-the-art.

View PDFchevron_right
A multimodal feature learning approach for sentiment analysis of social network multimedia
Marco Bertini

Multimedia Tools and Applications, 2015

In this paper we investigate the use of a multimodal feature learning approach, using neural network based models such as Skip-gram and Denoising Autoencoders, to address sentiment analysis of micro-blogging content, such as Twitter short messages, that are composed by a short text and, possibly, an image. The approach used in this work is motivated by the recent advances in: i) training language models based on neural networks that have proved to be extremely efficient when dealing with web-scale text corpora, and have shown very good performances when dealing with syntactic and semantic word similarities; ii) unsupervised learning, with neural networks, of robust visual features, that are recoverable from partial observations that may be due to occlusions or noisy and heavily modified images. We propose a novel architecture that incorporates these neural networks, testing it on several standard Twitter datasets, and showing that the approach is efficient and obtains good classification results.

View PDFchevron_right

Related topics

  • Computer Science

    • Cited by

    Adversarial Examples Detection in Features Distance Spaces
    Fabrizio Falchi

    Lecture Notes in Computer Science

    Maliciously manipulated inputs for attacking machine learning methods-in particular deep neural networks-are emerging as a relevant issue for the security of recent artificial intelligence technologies, especially in computer vision. In this paper, we focus on attacks targeting image classifiers implemented with deep neural networks, and we propose a method for detecting adversarial images which focuses on the trajectory of internal representations (i.e. hidden layers neurons activation, also known as deep features) from the very first, up to the last. We argue that the representations of adversarial inputs follow a different evolution with respect to genuine inputs, and we define a distance-based embedding of features to efficiently encode this information. We train an LSTM network that analyzes the sequence of deep features embedded in a distance space to detect adversarial examples. The results of our preliminary experiments are encouraging: our detection scheme is able to detect adversarial inputs targeted to the ResNet-50 classifier pretrained on the ILSVRC'12 dataset and generated by a variety of crafting algorithms.

    View PDFchevron_right
    Detection of extreme sentiments on social networks with BERT
    Sebastião Pais

    Social Network Analysis and Mining

    Online social networking platforms allow people to freely express their ideas, opinions, and emotions negatively or positively. Previous studies have examined sentiments on these platforms to study their behavior in different contexts and purposes. The mechanism of collecting public opinion information has attracted researchers to automatically classify the polarity of public opinions based on the use of concise language in messages, such as tweets, by analyzing social media data. In this paper, we extend the preceding work where an unsupervised approach to automatically detect extreme opinions/posts in social networks is proposed. The performance of the proposed approach is evaluated on five different social network and media datasets. In this work, we use a semi-supervised approach known as BERT to reevaluate the accuracy of our prior approach and the obtained classified dataset. The experiment proves that in these datasets, posts that were previously classified as negative or pos...

    View PDFchevron_right
    The InsightsNet Climate Change Corpus (ICCC)
    Elena Volkanovska

    Datenbank-Spektrum

    The discourse on climate change has become a centerpiece of public debate, thereby creating a pressing need to analyze the multitude of messages created by the participants in this communication process. In addition to text, information on this topic is conveyed multimodally, through images, videos, tables and other data objects that are embedded within documents and accompany the text. This paper presents the process of building a multimodal pilot corpus to the InsightsNet Climate Change Corpus (ICCC) and using natural language processing (NLP) tools to enrich corpus (meta)data, thus creating a dataset that lends itself to the exploration of the interplay between the various modalities that constitute the discourse on climate change. We demonstrate how the pilot corpus can be queried for relevant information in two types of databases, and how the proposed data model promotes a more comprehensive sentiment analysis approach.

    View PDFchevron_right
    Interpretable Multimodal Sentiment Classification Using Deep Multi-View Attentive Network of Image and Text Data
    mohammad asadpour

    IEEE Access

    Multimodal data can convey user emotions and feelings more effectively and interactively than unimodal content. Thus, multimodal sentiment analysis (MSA) research has recently acquired great significance as a field of study. However, most current approaches either acquire sentimental features independently for each modality or simply combine multiple modal features. Thus, semantic details pertinent to sentiment analysis and the relationship between visual and textual content are neglected. Furthermore, most available multimodal datasets are sentiment-annotated, although user emotions are usually rich and unlimited. Motivated by these observations, this paper proposes a novel deep multi-view attentive network (DMVAN) for robust multimodal sentiment and emotion classification. The DMVAN model has three phases: feature learning, attentive interaction learning, and cross-modal fusion learning. During the feature learning phase, visual features from a multi-view perspective (region and scene) and textual features from various levels of analysis (word, sentence, and document) are extracted to capture information effectively for accurate classification. In the attentive interaction learning phase, the image-text interaction learning mechanism is employed to enhance visual and textual information interaction by extracting sentimental and discriminative visual features and utilizing the textual information to guide the learning process of image features. Moreover, a cross-modal fusion learning module is developed to incorporate different features into a comprehensive framework that takes advantage of the complementary aspects of multiple modalities. Then, a multi-head attention mechanism is employed to extract and merge sufficient data from the intermediate features, thereby aiding in developing a robust joint representation. Finally, a multi-layer perceptron with multiple stacking-fully connected layers is used to deeply fuse the modal features, thereby enhancing sentiment classification performance. An interpretable multimodal sentiment classification model is further developed utilizing the local interpretable model-agnostic explanation model (LIME) to ensure the model's explainability and strength. To perform a multimodal emotion classification, an image-text emotion dataset named Emotion-Getty (EMO-G) is constructed from Getty Images and labeled by distinct emotions. The proposed model is tested on three real-world datasets, attaining 99.801% accuracy on Binary_Getty (BG), 96.867% on Twitter, and 96.174% on the EMO-G dataset. These results show that the suggested model outperforms single-model techniques and current state-of-the-art methodologies based on model evaluation criteria.

    View PDFchevron_right
    580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved