Academia.eduAcademia.edu

Figure 11 – uploaded by Firoj Alam

See full PDFdownloadDownload figure
Table 11: Class label mapping and grouping for CrisisLex, CrisisNLP, ISCRAM2013, and SWDM2013 datasets.

Table 11 Class label mapping and grouping for CrisisLex, CrisisNLP, ISCRAM2013, and SWDM2013 datasets.

Related Figures (16)

Different from them, Strassel et al. (2017) defines categories based on need types (e.g., evacuation, food supply) and issue type (e.g., civil unrest). In this study, we use the class labels that are highly important for humanitarian aid for disaster response task, which also has a commonality across the publicly available resources. Classification Algorithms: Despite a majority of studies in crisis informatics literature employ tradi- tional machine learning algorithms for automatic event detection, event type classification, and fine-grained humanitarian information type classification, several recent works explore deep learning algorithms in disaster-related tweet classification tasks. The study of Nguyen et al. (2017) and Neppalli et al. (2018) perform comparative experiments between different classical and deep learning algorithms including Support Vector Machines (SVM), Logistic Regression (LR), Random Forests (RF), Recurrent Neural Networks (RNN), and Convolutional Neural Networks (CNN). Their experimental results suggest that CNN outperforms other algorithms. Though in another study, Burel and Alani (2018) reports that SVM and CNN can provide very competitive results in some cases. CNNs have also been explored in event type- specific filtering model (Kersten et al., 2019) and few-shot learning (Kruspe et al., 2019). Very recently different types of embedding representations have been proposed in literature such as Embeddings from Language Models (ELMo) (Peters et al., 2018), Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2018), and XLNet (Yang et al., 2019) for different NLP tasks. For disaster-related classification, Jain et al. (2019) investigates these embedding representations and achieves similar results. Classification Algorithms: Despite a majority of studies in crisis informatics literature employ tradi-
Different from them, Strassel et al. (2017) defines categories based on need types (e.g., evacuation, food supply) and issue type (e.g., civil unrest). In this study, we use the class labels that are highly important for humanitarian aid for disaster response task, which also has a commonality across the publicly available resources. Classification Algorithms: Despite a majority of studies in crisis informatics literature employ tradi- tional machine learning algorithms for automatic event detection, event type classification, and fine-grained humanitarian information type classification, several recent works explore deep learning algorithms in disaster-related tweet classification tasks. The study of Nguyen et al. (2017) and Neppalli et al. (2018) perform comparative experiments between different classical and deep learning algorithms including Support Vector Machines (SVM), Logistic Regression (LR), Random Forests (RF), Recurrent Neural Networks (RNN), and Convolutional Neural Networks (CNN). Their experimental results suggest that CNN outperforms other algorithms. Though in another study, Burel and Alani (2018) reports that SVM and CNN can provide very competitive results in some cases. CNNs have also been explored in event type- specific filtering model (Kersten et al., 2019) and few-shot learning (Kruspe et al., 2019). Very recently different types of embedding representations have been proposed in literature such as Embeddings from Language Models (ELMo) (Peters et al., 2018), Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2018), and XLNet (Yang et al., 2019) for different NLP tasks. For disaster-related classification, Jain et al. (2019) investigates these embedding representations and achieves similar results. Classification Algorithms: Despite a majority of studies in crisis informatics literature employ tradi-
Figure 1: Number of near-duplicates in different bins obtained from consolidated informativeness tweet after label mapping. Tweets will lower similarity (< 0.7) bins are not reported here.
Figure 1: Number of near-duplicates in different bins obtained from consolidated informativeness tweet after label mapping. Tweets will lower similarity (< 0.7) bins are not reported here.
Figure 2: Exact- and near-duplicates in informativeness tweets. Number on top of each bar represent total number and the number in the parenthesis represent percentage of consolidated exact and near duplicates from the respective dataset. datasets comparatively. For each of these datasets, we wanted to see which event’s duplicates appear most. In CrisisLex, the majority of the exact duplicates appear in “Queensland floods (2013)! consisting of 2270 exact duplicates. The second majority is “West Texas explosion 92013)” event, which consists of 1301 duplicates. Compared to CrisisLex, the exact duplicates are low in CrisisNLP, and the majority of such duplicates appear in the “Philippines Typhoon Hagupit (2014)” event with 1084 tweets. For the humanitarian tweets, we observed similar characteristics of Figure 2.
Figure 2: Exact- and near-duplicates in informativeness tweets. Number on top of each bar represent total number and the number in the parenthesis represent percentage of consolidated exact and near duplicates from the respective dataset. datasets comparatively. For each of these datasets, we wanted to see which event’s duplicates appear most. In CrisisLex, the majority of the exact duplicates appear in “Queensland floods (2013)! consisting of 2270 exact duplicates. The second majority is “West Texas explosion 92013)” event, which consists of 1301 duplicates. Compared to CrisisLex, the exact duplicates are low in CrisisNLP, and the majority of such duplicates appear in the “Philippines Typhoon Hagupit (2014)” event with 1084 tweets. For the humanitarian tweets, we observed similar characteristics of Figure 2.
Table 2: Examples of near-duplicates with similarity scores selected from informativeness tweets. Dupli- cates are highlighted. Sim. refers to similarity value. Dup. refers to whether we consider them as duplicate and filtered. The symbol (X) indicates a duplicate, which we dropped and the symbol (W) indicates a not duplicate, which we have included in our dataset.
Table 2: Examples of near-duplicates with similarity scores selected from informativeness tweets. Dupli- cates are highlighted. Sim. refers to similarity value. Dup. refers to whether we consider them as duplicate and filtered. The symbol (X) indicates a duplicate, which we dropped and the symbol (W) indicates a not duplicate, which we have included in our dataset.
Table 3: Data distribution of informativeness across different sources the imbalance in class distribution seems to persist (last column in Table 4). For example, the distribution of “Not humanitarian” is relatively higher (37.40%) than other class labels, which might have to be under-sampled for training the classification model. In Table 4, we highlighted some class labels, which we dropped in the rest of the classification experiments conducted in this study, however, tweets with those class labels will be available in the released datasets. The reason for not including them in the experiments is that we aim to develop classifiers for the disaster response tasks only.
Table 3: Data distribution of informativeness across different sources the imbalance in class distribution seems to persist (last column in Table 4). For example, the distribution of “Not humanitarian” is relatively higher (37.40%) than other class labels, which might have to be under-sampled for training the classification model. In Table 4, we highlighted some class labels, which we dropped in the rest of the classification experiments conducted in this study, however, tweets with those class labels will be available in the released datasets. The reason for not including them in the experiments is that we aim to develop classifiers for the disaster response tasks only.
Table 4: Data distribution of humanitarian categories across different datasets.
Table 4: Data distribution of humanitarian categories across different datasets.
Event-aware Training The availability of annotated data for a disaster event is usually scarce. One of the advantages of our compiled data is to have identical classes across several disaster events. This enables us to combine the annotated data from all previous disasters for the classification. Though this increases the size of the training data substantially, the classifier may result in sub-optimal performance due to the inclusion of heterogeneous data (i.e., a variety of disaster types and occurs in a different part of the world).
Event-aware Training The availability of annotated data for a disaster event is usually scarce. One of the advantages of our compiled data is to have identical classes across several disaster events. This enables us to combine the annotated data from all previous disasters for the classification. Though this increases the size of the training data substantially, the classifier may result in sub-optimal performance due to the inclusion of heterogeneous data (i.e., a variety of disaster types and occurs in a different part of the world).
Table 7: Classification results of the consolidated dataset using CNN and BERT.
Table 7: Classification results of the consolidated dataset using CNN and BERT.
perform also consistently better in both precision and recall. In Table 8, we report class-wise performance of both CNN and BERT models for the humanitarian task. BERT performs better than or on par with CNN across all classes. More importantly, BERT performs substantially better than CNN in the case of minority classes as highlighted in the table.
perform also consistently better in both precision and recall. In Table 8, we report class-wise performance of both CNN and BERT models for the humanitarian task. BERT performs better than or on par with CNN across all classes. More importantly, BERT performs substantially better than CNN in the case of minority classes as highlighted in the table.
Table 9: Classification results of the event-aware experiments using the consolidated dataset.
Table 9: Classification results of the event-aware experiments using the consolidated dataset.
Table 10: Events in CrisisLex, CrisisNLP, ISCRAM2013 and SWDM2013 datasets.
Table 10: Events in CrisisLex, CrisisNLP, ISCRAM2013 and SWDM2013 datasets.
Table 14: Class label mapping for CrisisMMD.
Table 14: Class label mapping for CrisisMMD.
Related topics:
Computer ScienceInformaticsData ScienceScripting LanguageHumanitarian Crisis
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved