Floods Detection in Twitter Text and Images

2019

The aim of this study is to detect flooding events by analyzing both texts published by African online news outlets as well as the accompanying article images. The data is provided by MediaEval 2019 within the Multimedia Satellite Task. Our contributions are related to the imageand text-based subtasks. In order to solve the required classification subtasks, we build models capable to extract features from images and texts separately, and then combine them to obtain a complex classifier, providing a better evidence of flooding. Specifically, we adopt the MobileNet architecture which is based on convolutional layers for image processing and also employ a robust text processing method based on long short-term memory cells. The results of our final models on the official test sets are promising, 85.26% average F1-score on the first subtask and 66.19% average F1-score on the second subtask.

2017

In this paper we propose a multi-modal deep learning approach to detect floods in social media posts. Social media posts normally contain somemetadata and/or visual information, therefore in order to detect the floods we use this information. The model is based on a Convolutional Neural Network which extracts the visual features and a bidirectional Long Short-Term Memory network to extract the semantic features from the textual metadata. We validate the method on images extracted from Flickr which contain both visual information and metadata and compare the results when using both, visual information only or metadata only. This work has been done in the context of the MediaEval Multimedia Satellite Task.

2018

In this paper, we present an approach to identification of the images that depict passable and non-passable roads, from a collection of flood-related tweet images. Our key insight is that the local information from domain-specific concepts (‘boat’, ‘person’ and ‘car’) can be exploited to help determine whether an image depicts a location that is passable. We use concept detection as the basis for features that encode local information. We use conventional features, i.e., presence of concepts and visual features extracted from the concept region, but also a novel light-weight feature, i.e., the aspect ratio of the bounding box. Experimental results show that integrating local semantic information yields slightly better performance than only using image-level CNN representation. Text features are not competitive.

2020

Social media is the main source of providing information in the form of user generated content(UGC) on the severity of Natural disaster. However, extracting the relevant information in an organized manner from social media has been a challenging task. Therefore, the purpose of this research paper is to provide an efficient algorithm that can reduce the workforce of disaster management by classifying relevant social media streams, humanitarian aid, and damage assessment. The project will classify relevant tweets of hurricane harvey, hurricane irma, california wildfire, mexico earthquake, Nepal earthquake, and iran-iraq earthquake by applying Natural language processing and computer vision based deep learning models ensemble together. The first task will include the categorization of tweets and their respective images on the basis of their relevance(containing information about natural disasters). Secondly, the image data will be categorized further based on humanitarian aid which includes injured or dead people, infrastructure damage, vehicle damage, and missing or found people. Finally, the damage assessment of the events will be categorized based on mild, moderate, and severe. The demo application will be developed to provide a user interface for natural disaster management teams to access disaster.

ISCRAM 2021 Conference Proceedings – 18th International Conference on Information Systems for Crisis Response and Management, 2021

Social media applications such as Twitter and Facebook are fast becoming a key instrument in gaining situational awareness (understanding the bigger picture of the situation) during disasters. This has provided multiple opportunities to gather relevant information in a timely manner to improve disaster response. In recent years, identifying crisis-related social media posts is analysed as an automatic task using machine learning (ML) or deep learning (DL) techniques. However, such supervised learning algorithms require labelled training data in the early hours of a crisis. Recently, multiple manually labelled disaster-related open-source twitter datasets have been released. In this work, we collected 192, 948 tweets by combining a number of such datasets, preprocessed, filtered and duplicate removed, which resulted in 117, 954 tweets. Then we evaluated the performance of multiple ML and DL algorithms in classifying disaster-related tweets in three settings, namely "in-disaster", "out-disaster" and "cross-disaster". Our results show that the Bidirectional LSTM model with Word2Vec embeddings performs well for the tweet classification task in all three settings. We also make available the preprocessing steps and trained weights for future research.

Mathematics

Social media platforms such as Twitter are a vital source of information during major events, such as natural disasters. Studies attempting to automatically detect textual communications have mostly focused on machine learning and deep learning algorithms. Recent evidence shows improvement in disaster detection models with the use of contextual word embedding techniques (i.e., transformers) that take the context of a word into consideration, unlike the traditional context-free techniques; however, studies regarding this model are scant. To this end, this paper investigates a selection of ensemble learning models by merging transformers with deep neural network algorithms to assess their performance in detecting informative and non-informative disaster-related Twitter communications. A total of 7613 tweets were used to train and test the models. Results indicate that the ensemble models consistently yield good performance results, with F-score values ranging between 76% and 80%. Simp...

2020

This paper presents the method proposed and implemented by team FAST-NU-DS, in "The Flood-related Multimedia Task at MediaEval 2020". The task includes data of tweets in Italian language, extracted during floods between 2017 and 2019. The proposed method has utilized text of the tweet and its relevant image for the purpose of binary classification, which identifies whether or not the particular tweet is about flood incident. The proposed method has designed an ensemble based method for the classification of tweets, on the basis of textual data, visual data and combination of both. For visual data, the proposed method has utilized the technique of data augmentation for oversampling of the minority class and applied stratified random sampling for the selection of input. Moreover, Visual Geometry Group (VGG16) convolutional neural network, pretrained on ImageNet and Places365 is utilized by the proposed method. For classification of textual data, the technique of Term Frequen...

IEEE Access, 2022

During the past decade, social media platforms have been extensively used for information dissemination by the affected community and humanitarian agencies during a disaster. Although many studies have been done recently to classify the informative and non-informative messages from social media posts, most are unimodal, i.e., have independently used textual or visual data to build deep learning models. In the present study, we integrate the complementary information provided by the text and image messages about the same event posted by the affected community on the social media platform Twitter and build a multimodal deep learning model based on the concept of the attention mechanism. The attention mechanism is a recent breakthrough that has revolutionized the field of deep learning. Just as humans pay more attention to a specific part of the text or image, ignoring the rest, neural networks can also be trained to concentrate on more relevant features through the attention mechanism. We propose a novel Cross-Attention Multi-Modal (CAMM) deep neural network for classifying multimodal disaster data, which uses the attention mask of the textual modality to highlight the features of the visual modality. We compare CAMM with unimodal models and the most popular bilinear multimodal models, MUTAN and BLOCK, generally used for visual question answering. CAMM achieves an average F1-score of 84.08%, better than the MUTAN and BLOCK methods by 6.31% and 5.91%, respectively. The proposed cross-attention-based multimodal deep learning method outperforms the current state-of-the-art fusion methods on the benchmark multimodal disaster dataset by highlighting more relevant cross-domain features of text and image tweets. INDEX TERMS Deep convolutional neural network (DCNN), disaster management, multimodal learning, attention mechanism, cross-attention, social media, Twitter. I. INTRODUCTION 20 In the past decade, emergency managers and safety organi-21 zations have started using social media platforms to share 22 critical information for planning and implementing rescue 23 operations during a disaster. The decision-makers utilize 24 timely, first-hand, and location-based messages posted by 25 eyewitnesses on social media platforms to deploy resources 26 and enhance their response efforts. Innovative use of these 27 platforms allows humanitarian teams to engage directly with 28 the affected public during all phases of disaster manage-29 ment. Among several social media platforms, Twitter is most 30 prevalent during natural disasters [1], [2]. Twitter text mes-31 sages, called tweets that consist of up to 280 characters, give 32 747

Electronics, 2021

A flood is an overflow of water that swamps dry land. The gravest effects of flooding are the loss of human life and economic losses. An early warning of these events can be very effective in minimizing the losses. Social media websites such as Twitter and Facebook are quite effective in the efficient dissemination of information pertinent to any emergency. Users on these social networking sites share both textual and rich content images and videos. The Multimedia Evaluation Benchmark (MediaEval) offers challenges in the form of shared tasks to develop and evaluate new algorithms, approaches and technologies for explorations and exploitations of multimedia in decision making for real time problems. Since 2015, the MediaEval has been running a shared task of predicting several aspects of flooding and through these shared tasks, many improvements have been observed. In this paper, the classification framework VRBagged-Net is proposed and implemented for flood classification. The frame...

AM Publications,India, 2025

Every activity in disaster management demands accurate and up-to-date information to allow a quick, easy, and cost-effective response to reduce the possible loss of lives and properties. It is a challenging and complex task to acquire information from different regions of a disaster-affected area in a timely fashion. The extensive spread and reach of social media and networks such as Twitter allow people to share information in real-time. However, gathering of valuable information requires a series of operations such as (1) processing each tweet for the text classification, (2) possible location determination of people needing help based on tweets, and (3) priority calculations of rescue tasks based on the classification of tweets. These are three primary challenges in developing an effective rescue scheduling operation using social media data. In this paper, first, we propose a deep learning model combining attention based Bi-directional Long Short-Term Memory (BLSTM) and Convolutional Neural Network (CNN) to classify the tweets. Next, we perform feature engineering to create an auxiliary feature map which dramatically increases the model accuracy. In our experiments using data from Hurricanes Harvey and Irma, it is observed that our proposed approach performs better compared to other classification methods based on Precision, Recall, F1-score, and Accuracy, and is highly effective to determine the priority of a tweet.