Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Literature Review
Dataset Description
Methodology
Data and Annotation
Experimental Setting
Results
Analysis
Conclusion and Future Work
References
All Topics
Computer Science
Machine Learning

Claim Detection in Persian Twitter Posts

Profile image of IJICTR JournalIJICTR Journal

2024, International Journal of Information and Communication Technology Research

Abstract

The proliferation of false information on social media has profound negative impacts across various aspects of people's lives. To mitigate these effects, numerous studies have focused on developing automated factchecking systems aimed at enhancing the accuracy and reliability of news and information. Claim detection, recognized as the initial stage in constructing such systems, has been explored in several languages. In our paper, we introduce a corpus of Persian tweets annotated with 11 labels derived from linguistic analysis, representing different types of claims. Additionally, we establish a baseline claim detection model to assess the dataset. This study frames claim detection as a classification task and employs a transformer-based approach to train a multi-label classifier capable of identifying various types of claims in Persian texts.

Related papers

ClaimHunter: An unattended tool for automated claim detection on Twitter
Irene Larraz

KnOD@WWW, 2021

As political campaigns have moved from traditional media to social networks, fact-checkers must also adapt how they are working. The explosion of information (and disinformation) on social networks makes impossible to manually fact-check each piece of data. With this reality in mind, Newtral, a fact-checking organization, has developed its own automated monitor tool for Twitter: ClaimHunter. Recently, deep learning approaches have obtained very high performance across many different NLP tasks. Automated claim detection is not an exception. These models are showing promising results on fact-checking scenarios without task-specific feature engineering. Based on the BERT architecture, ClaimHunter AI models shown a 80% F1 score tested on real-life scenarios with expert fact-checkers. Through a simple UI interface deployed on Slack, ClaimHunter notifies journalists and gets feedback from their day-today work to improve the final performance of the algorithm. Launched 6 months ago, ClaimHunter has processed more than 130.000 tweets expanding Newtral's operation beyond national politicians to the regional and local level. The number of reviewed claims per day have increased by a multiplicative factor of 10 since the adoption of the ClaimHunter tool by Newtral's fact-checking team. This paper focuses on explaining the challenges of building such a system inside of a fact-checking organization: data labelling and fact-checker alignment, system architecture, testing and deployment of underlying models, continuous feedback and model refinement. Full article: https://ceur-ws.org/Vol-2877/paper3.pdf

View PDFchevron_right
Overview of the CLEF-2021 CheckThat! Lab Task 2 on Detecting Previously Fact-Checked Claims in Tweets and Political Debates
Firoj Alam

2021

We describe the fourth edition of the CheckThat! Lab, part of the 2021 Conference and Labs of the Evaluation Forum (CLEF). The lab evaluates technology supporting three tasks related to factuality, and it covers Arabic, Bulgarian, English, Spanish, and Turkish. Here, we present the task 2, which asks to detect previously fact-checked claims (in two languages). A total of four teams participated in this task, submitted a total of sixteen runs, and most submissions managed to achieve sizable improvements over the baselines using transformer based models such as BERT, RoBERTa. In this paper, we describe the process of data collection and the task setup, including the evaluation measures used, and we give a brief overview of the participating systems. Last but not least, we release to the research community all datasets from the lab as well as the evaluation scripts, which should enable further research in detecting previously fact-checked claims. © 2021 Copyright for this paper by its ...

View PDFchevron_right
Overview of CheckThat! 2020i Arabic: Automatic Identification and Verification of Claims in Social Media
Reem Suwaileh

2020

In this paper, we present an overview of the Arabic tasks of the third edition of the CheckThat! Lab at CLEF 2020. The lab featured three Arabic tasks over social media (and the Web): Task 1 on checkworthiness estimation, Task 3 on evidence retrieval, and Task 4 on claim verification. For evaluation, we collected a dataset of Arabic tweets and Web pages consisting of 7.5K tweets and 14,742 Web pages. The systems in the ranking tasks (Task 1 and Task 3) were evaluated using precision at 30 (P@30) and precision at 10 (P@10), respectively. F1 was the official evaluation measure for Task 4. Eight teams submitted runs to the Arabic tasks, which is double the number of teams participating in the Arabic tasks of the CheckThat! lab at CLEF 2019. The most successful approach to Task 1 used an Arabic pre-trained language model, while text similarity measures and linguistic features were used in the other tasks. We release to the research community all datasets from the lab, which should enabl...

View PDFchevron_right
bigIR at CLEF 2019: Automatic Verification of Arabic Claims over the Web
Fatima Haouari

2019

With the proliferation of fake news and its prevalent impact on democracy, journalism, and public opinions, manual fact-checkers become unscalable to the volume and speed of fake news propagation. Automatic fact-checkers are therefore needed to prevent the negative impact of fake news in a fast and effective way. In this paper, we present our participation in Task 2 of CLEF-2019 CheckThat! Lab, which addresses the problem of finding evidence over the Web for verifying Arabic claims. We participated in all of the four subtasks and adopted a machine learning approach in each with different set of features that are extracted from both the claim and the corresponding retrieved Web search result pages. Our models, trained solely over the provided training data, for the different subtasks exhibited relatively-good performance. Our official results, on the testing data, show that our best performing runs achieved the best overall performance in subtasks A and B among 7 and 8 participating ...

View PDFchevron_right
CheckThat! at CLEF 2020: Enabling the Automatic Identification and Verification of Claims in Social Media
Reem Suwaileh

Advances in Information Retrieval, 2020

We describe the third edition of the CheckThat! Lab, which is part of the 2020 Cross-Language Evaluation Forum (CLEF). CheckThat! proposes four complementary tasks and a related task from previous lab editions, offered in English, Arabic, and Spanish. Task 1 asks to predict which tweets in a Twitter stream are worth fact-checking. Task 2 asks to determine whether a claim posted in a tweet can be verified using a set of previously fact-checked claims. Task 3 asks to retrieve text snippets from a given set of Web pages that would be useful for verifying a target tweet's claim. Task 4 asks to predict the veracity of a target tweet's claim using a set of Web pages and potentially useful snippets in them. Finally, the lab offers a fifth task that asks to predict the check-worthiness of the claims made in English political debates and speeches. CheckThat! features a full evaluation framework. The evaluation is carried out using mean average precision or precision at rank k for ranking tasks, and F1 for classification tasks.

View PDFchevron_right
Overview of CheckThat! 2020: Automatic Identification and Verification of Claims in Social Media
Alex Nikolov

Lecture Notes in Computer Science, 2020

We present an overview of the third edition of the CheckThat! Lab at CLEF 2020. The lab featured five tasks in two different languages: English and Arabic. The first four tasks compose the full pipeline of claim verification in social media: Task 1 on check-worthiness estimation, Task 2 on retrieving previously fact-checked claims, Task 3 on evidence retrieval, and Task 4 on claim verification. The lab is completed with Task 5 on check-worthiness estimation in political debates and speeches. A total of 67 teams registered to participate in the lab (up from 47 at CLEF 2019), and 23 of them actually submitted runs (compared to 14 at CLEF 2019). Most teams used deep neural networks based on BERT, LSTMs, or CNNs, and achieved sizable improvements over the baselines on all tasks. Here we describe the tasks setup, the evaluation results, and a summary of the approaches used by the participants, and we discuss some lessons learned. Last but not least, we release to the research community all datasets from the lab as well as the evaluation scripts, which should enable further research in the important tasks of check-worthiness estimation and automatic claim verification.

View PDFchevron_right
Unsupervised corpus–wide claim detection
Benjamin Sznajder

Proceedings of the 4th Workshop on Argument Mining

Automatic claim detection is a fundamental argument mining task that aims to automatically mine claims regarding a topic of consideration. Previous works on mining argumentative content have assumed that a set of relevant documents is given in advance. Here, we present a first corpuswide claim detection framework, that can be directly applied to massive corpora. Using simple and intuitive empirical observations, we derive a claim sentence query by which we are able to directly retrieve sentences in which the prior probability to include topic-relevant claims is greatly enhanced. Next, we employ simple heuristics to rank the sentences, leading to an unsupervised corpus-wide claim detection system, with precision that outperforms previously reported results on the task of claim detection given relevant documents and labeled data.

View PDFchevron_right
Accenture at CheckThat! 2020: If you say so: Post-hoc fact-checking of claims using transformer-based models
Paul Rodrigues

Working Notes of CLEF 2020-Conference and Labs of the Evaluation Forum, 2020

We introduce the strategies used by the Accenture Team for the CLEF2020 CheckThat! Lab, Task 1, on English and Arabic. This shared task evaluated whether a claim in social media text should be professionally fact checked. To a journalist, a statement presented as fact, which would be of interest to a large audience, requires professional fact-checking before dissemination. We utilized BERT and RoBERTa models to identify claims in social media text a professional fact-checker should review, and rank these in priority order for the fact-checker. For the English challenge, we fine-tuned a RoBERTa model and added an extra mean pooling layer and a dropout layer to enhance generalizability to unseen text. For the Arabic task, we fine-tuned Arabic-language BERT models and demonstrate the use of back-translation to amplify the minority class and balance the dataset. The work presented here was scored 1st place in the English track, and 1st, 2nd, 3rd, and 4th place in the Arabic track.

View PDFchevron_right
Overview of the CLEF-2019 CheckThat!: Automatic Identification and Verification of Claims
Reem Suwaileh

2021

We present an overview of the second edition of the CheckThat! Lab at CLEF 2019. The lab featured two tasks in two different languages: English and Arabic. Task 1 (English) challenged the participating systems to predict which claims in a political debate or speech should be prioritized for fact-checking. Task 2 (Arabic) asked to (A) rank a given set of Web pages with respect to a check-worthy claim based on their usefulness for fact-checking that claim, (B) classify these same Web pages according to their degree of usefulness for fact-checking the target claim, (C) identify useful passages from these pages, and (D) use the useful pages to predict the claim’s factuality. CheckThat! provided a full evaluation framework, consisting of data in English (derived from fact-checking sources) and Arabic (gathered and annotated from scratch) and evaluation based on mean average precision (MAP) and normalized discounted cumulative gain (nDCG) for ranking, and F1 for classification. A total of...

View PDFchevron_right
Towards Automated Factchecking: Developing an Annotation Schema and Benchmark for Consistent Automated Claim Detection
Mevan Babakar

ArXiv, 2018

In an effort to assist factcheckers in the process of factchecking, we tackle the claim detection task, one of the necessary stages prior to determining the veracity of a claim. It consists of identifying the set of sentences, out of a long text, deemed capable of being factchecked. This paper is a collaborative work between Full Fact, an independent factchecking charity, and academic partners. Leveraging the expertise of professional factcheckers, we develop an annotation schema and a benchmark for automated claim detection that is more consistent across time, topics and annotators than previous approaches. Our annotation schema has been used to crowdsource the annotation of a dataset with sentences from UK political TV shows. We introduce an approach based on universal sentence representations to perform the classification, achieving an F1 score of 0.83, with over 5% relative improvement over the state-of-the-art methods ClaimBuster and ClaimRank. The system was deployed in produc...

View PDFchevron_right

References (84)

  1. Herman, Edward S., and Noam Chomsky. Manufacturing con- sent: The political economy of the mass media. Random House, 2010.
  2. Bradshaw, Samantha, and Philip N. Howard. "The global dis- information order: 2019 global inventory of organised social media manipulation." (2019).
  3. Shu, Kai, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. "Fake news detection on social media: A data mining per- spective." ACM SIGKDD explorations newsletter 19, no. 1 (2017): 22-36.
  4. Zubiaga, Arkaitz, Ahmet Aker, Kalina Bontcheva, Maria Lia- kata, and Rob Procter. "Detection and resolution of rumours in social media: A survey." ACM Computing Surveys (CSUR) 51, no. 2 (2018): 1-36.
  5. Bastick, Zach. "Would you notice if fake news changed your behavior? An experiment on the unconscious effects of disin- formation." Computers in human behavior 116 (2021): 106633.
  6. Ecker, Ullrich KH, Stephan Lewandowsky, John Cook, Philipp Schmid, Lisa K. Fazio, Nadia Brashier, Panayiota Kendeou, Emily K. Vraga, and Michelle A. Amazeen. "The psychologi- cal drivers of misinformation belief and its resistance to cor- rection." Nature Reviews Psychology 1, no. 1 (2022): 13-29.
  7. Plotnick, Linda, Starr Hiltz, Sukeshini Grandhi, and Julie Dug- dale. "Real or fake? User behavior and attitudes related to de- termining the veracity of social media posts." arXiv preprint arXiv:1904.03989 (2019).
  8. Featherstone, Jieyu Ding, C. A. Davis, and J. Zhang. "Correct- ing Vaccine Misinformation on Social Media Using Fact- checking Labels."
  9. In APHA's 2019 annual meeting and expo. 2019.
  10. Roozenbeek, Jon, Claudia R. Schneider, Sarah Dryhurst, John Kerr, Alexandra LJ Freeman, Gabriel Recchia, Anne Marthe Van Der Bles, and Sander Van Der Linden. "Susceptibility to misinformation about COVID-19 around the world." Royal So- ciety open science 7, no. 10 (2020): 201199.
  11. Greene, Ciara M., and Gillian Murphy. "Quantifying the ef- fects of fake news on behavior: Evidence from a study of COVID-19 misinformation." Journal of Experimental Psychol- ogy: Applied (2021).
  12. Zhou, Cheng, Haoxin Xiu, Yuqiu Wang, and Xinyao Yu. "Characterizing the dissemination of misinformation on social media in health emergencies: An empirical study based on COVID-19." Information Processing and Management 58, no. 4 (2021): 102554.
  13. Van Der Linden, Sander, Costas Panagopoulos, and Jon Roozenbeek. "You are fake news: political bias in perceptions of fake news." Media, Culture and Society 42, no. 3 (2020): 460-470.
  14. Muqsith, Munadhil Abdul, R. Ridho Pratomo, Anna Gustina Zaina, and Ana Kuswanti. "Fake News as a Tool to Manipulate the Public with False Information." In 2nd International Indo- nesia Conference on Interdisciplinary Studies (IICIS 2021), pp. 118-127. Atlantis Press, 2021.
  15. Polak, Mateusz. "The misinformation effect in financial mar- kets: An emerging issue in behavioural fianance." e-Finanse: Financial Internet Quarterly 8, no. 3 (2012): 55-61.
  16. Kogan, Shimon, Tobias J. Moskowitz, and Marina Niessner. "Fake news: Evidence from financial markets." Available at SSRN 3237763 (2019).
  17. Rich, Patrick R., and Maria S. Zaragoza. "The continued inu- ence of implied and explicitly stated misinformation in news reports." Journal of experimental psychology: learning, memory, and cognition 42, no. 1 (2016): 62.
  18. Thorson, Emily. "Belief echoes: The persistent effects of cor- rected misinformation." Political Communication 33, no. 3 (2016): 460-480.
  19. Desai, Saoirse Connor, and Stian Reimers. "Some misinfor- mation is more easily countered: An experiment on the contin- ued inuence effect." In CogSci. 2018.
  20. Wang, Xuezhi, Cong Yu, Simon Baumgartner, and Flip Korn. "Relevant document discovery for fact-checking articles." In Companion Proceedings of the The Web Conference 2018, pp. 525-533. 2018.
  21. Jo, Saehan, Immanuel Trummer, Weicheng Yu, Xuezhi Wang, Cong Yu, Daniel Liu, and Niyati Mehta. "Verifying text sum- maries of relational data sets." In Proceedings of the 2019 In- ternational Conference on Management of Data, pp. 299-316. 2019.
  22. Trokhymovych, Mykola, and Diego Saez-Trumper. "Wikicheck: An end-to-end open source automatic fact-check- ing api based on wikipedia." In Proceedings of the 30th ACM International Conference on Information and Knowledge Man- agement, pp. 4155-4164. 2021.
  23. Graves, Lucas, Brendan Nyhan, and Jason Reier. "Understand- ing innovations in journalistic practice: A field experiment ex- amining motivations for fact-checking." Journal of Communi- cation 66, no. 1 (2016): 102-138.
  24. Graves, D. "Understanding the promise and limits of automated factchecking." (2018).
  25. Lippi, Marco, and Paolo Torroni. "Context-independent claim detection for argument mining." In Twenty-Fourth Interna- tional Joint Conference on Artificial Intelligence. 2015.
  26. Stab, Christian, and Iryna Gurevych. "Parsing argumentation structures in persuasive essays." Computational Linguistics 43, no. 3 (2017): 619-659.
  27. Konstantinovskiy, Lev, Oliver Price, Mevan Babakar, and Arkaitz Zubiaga. "Toward automated factchecking: Develop- ing an annotation schema and benchmark for consistent auto- mated claim detection." Digital Threats: Research and Practice 2, no. 2 (2021): 1-16.
  28. Shin, Jieun, Lian Jian, Kevin Driscoll, and Francois Bar. "The diffusion of misinformation on social media: Temporal pattern, message, and source." Computers in Human Behavior 83 (2018): 278-287.
  29. Wu, Liang, Fred Morstatter, Kathleen M. Carley, and Huan Liu. "Misinformation in social media: definition, manipulation, and detection." ACM SIGKDD Explorations Newsletter 21, no. 2 (2019): 80-90.
  30. Levy, Ran, Yonatan Bilu, Daniel Hershcovich, Ehud Aharoni, and Noam Slonim. "Context dependent claim detection." In Proceedings of COLING 2014, the 25th International Confer- ence on Computational Linguistics: Technical Papers, pp. 1489-1500. 2014.
  31. Lippi, Marco, and Paolo Torroni. "Argument mining: A ma- chine learning perspective." In International Workshop on The- ory and Applications of Formal Argumentation, pp. 163-176.
  32. Springer, Cham, 2015.
  33. Vlachos, Andreas, and Sebastian Riedel. "Fact checking: Task definition and dataset construction." In Proceedings of the ACL 2014 workshop on language technologies and computa- tional social science, pp. 18-22. 2014.
  34. Daniel, Anna, Terry Flew, and Christina Spurgeon. "The prom- ise of computational journalism." In Proceedings of the Aus- tralian and New Zealand Communication Association (ANZCA) Conference 2010: Media, Democracy and Change, pp. 1-19. Australia and New Zealand Communication Associ- ation, 2010.
  35. Cohen, Sarah, Chengkai Li, and Jun Yang. "C. Yu. Computa- tional journalism: A call to arms to database researchers." CIDR, 2011.
  36. Graves, D. "Understanding the promise and limits of automated factchecking." (2018).
  37. Guo, Zhijiang, Michael Schlichtkrull, and Andreas Vlachos. "A survey on automated fact-checking." Transactions of the Asso- ciation for Computational Linguistics 10 (2022): 178-206.
  38. Lazarski, Eric, Mahmood Al-Khassaweneh, and Cynthia How- ard. "Using NLP for Fact Checking: A Survey." Designs 5, no. 3 (2021)
  39. Walton, Douglas. "Argumentation theory: A very short intro- duction." In Argumentation in artificial intelligence, pp. 1-22.
  40. Springer, Boston, MA, 2009.
  41. Moens, Marie-Francine, Erik Boiy, Raquel Mochales Palau, and Chris Reed. "Automatic detection of arguments in legal texts." In Proceedings of the 11th international conference on Artifi- cial intelligence and law, pp. 225-230. 2007.
  42. Palau, Raquel Mochales, and Marie-Francine Moens. "Argu- mentation mining: the detection, classification and structure of arguments in text." In Proceedings of the 12th international conference on arti-ficial intelligence and law, pp. 98-107. 2009.
  43. Saint-Dizier, Patrick. "Processing natural language arguments with the< TextCoop> platform." Argument and Computation 3, no. 1 (2012): 49-82.
  44. Cabrio, Elena, and Serena Villata. "Natural language argu- ments: A combined approach." In ECAI 2012, pp. 205-210. IOS Press, 2012.
  45. Stylianou, Nikolaos, and Ioannis Vlahavas. "Transformed: End- to-End transformers for evidence-based medicine and argu- ment mining in medical literature." Journal of Biomedical In- formatics 117 (2021):103767.
  46. Al Khatib, Khalid, Tirthankar Ghosal, Yufang Hou, Anita deWaard, and Dayne Freitag. "Argument mining for scholarly document processing: Taking stock and looking ahead." In Proceedings of the Second Workshop on Scholarly Document Processing, pp. 56-65. 2021.
  47. Mebane, Waleed. "Detection of Claims and Supporting Evi- dence in Wikipedia Articles on Controversial Topics." PhD diss., 2017.
  48. Lawrence, John, and Chris Reed. "Combining argument mining techniques." In Proceedings of the 2nd Workshop on Argu- mentation Mining, pp. 127-136. 2015.17
  49. Budzynska, Katarzyna, Mathilde Janier, Juyeon Kang, Chris Reed, Patrick Saint-Dizier, Manfred Stede, and Olena Yaskorska. "Towards argument mining from dialogue." In Computational Models of Argument, pp. 185-196. IOS Press, 2014.
  50. Patwari, Ayush, Dan Goldwasser, and Saurabh Bagchi. "Tathya: A multi-classifier system for detecting check-worthy statements in political debates." In Proceedings of the 2017 ACM on Conference on Information and Knowledge Manage- ment, pp. 2259-2262. 2017.
  51. Hassan, Naeemul, Gensheng Zhang, Fatma Arslan, Josue Cara- ballo, Damian Jimenez, Siddhant Gawsane, Shohedul Hasan et al. "Claimbuster: The first-ever end-to-end fact-checking sys- tem." Proceedings of the VLDB Endowment 10, no. 12 (2017): 1945-1948.
  52. Gencheva, Pepa, Preslav Nakov, Liuis Marquez, Alberto Barron Cede, and Ivan Koychev. "A context-aware approach for de- tecting worth-checking claims in political debates." In Pro- ceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, pp. 267-276. 2017.
  53. Hanto, Vigdis, and Mats Tostrup. "Towards Automated Fake News Classification-On Building Collections for Claim Anal- ysis Research." Master's thesis, NTNU, 2018.
  54. Atanasova, Pepa, Preslav Nakov, Georgi Karadzhov, Mitra Mohtarami, and Giovanni Da San Martino. "Overview of the CLEF-2019 CheckThat! Lab: Automatic Identification and Verification of Claims. Task 1: Check-Worthiness." CLEF (Working Notes) 2380 (2019).
  55. Shaar, Shaden, Alex Nikolov, Nikolay Babulkov, Firoj Alam, Alberto Barron-Cedeno, Tamer Elsayed, Maram Hasanain et al. "Overview of CheckThat! 2020 English: Automatic identi- fication and verification of claims in social media." In CLEF (Working Notes). 2020.
  56. Atanasova, Pepa, Alberto Barron-Cedeno, Tamer Elsayed, Reem Suwaileh, Wajdi Zaghouani, Spas Kyuchukov, Giovanni Da San Martino, and Preslav Nakov. "Overview of the CLEF- 2018 CheckThat! Lab on automatic identification and verifica- tion of political claims. Task 1: Check-worthiness." arXiv pre- print arXiv:1808.05542 (2018).
  57. Barron-codeno, Alberto, Tamer Elsayed, Preslav Nakov, Gio- vanni Da San Martino, Maram Hasanain, Reem Suwaileh, Fat- ima Haouari et al. "Overview of CheckThat! 2020: Automatic identification and verification of claims in social media." In In- ternational Conference of the Cross-Language Evaluation Fo- rum for European Languages, pp. 215-236. Springer, Cham, 2020.
  58. Kartal, Yavuz Selim, and Mucahid Kutlu. "TrClaim-19: The first collection for Turkish check-worthy claim detection with annotator rationales." In Proceedings of the 24th Conference on Computational Natural Language Learning, pp. 386-395. 2020.
  59. Berendt, Bettina, Peter Burger, Rafael Hautekiet, Jan Jagers, Al- exander Pleijter, and Peter Van Aelst. "FactRank: Developing automated claim detection for Dutch-language fact-checkers." Online Social Networks and Media 22 (2021): 100113.
  60. Atanasova, Pepa, Preslav Nakov, Luis Marquez, Alberto Bar- ron-Cede~no, Georgi Karadzhov, Tsvetomila Mihaylova, Mi- tra Mohtarami, and James Glass. "Automatic fact-checking us- ing context and discourse information." Journal of Data and In- formation Quality (JDIQ) 11, no. 3 (2019): 1-27.18
  61. Williams, Evan, Paul Rodrigues, and Valerie Novak. "Accen- ture at CheckThat! 2020: If you say so: post-hoc fact-checking of claims using transformer-based models." arXiv preprint arXiv:2009.02431 (2020).
  62. Hasanain, Maram, and Tamer Elsayed. "bigIR at CheckThat! 2020: Multilingual BERT for Ranking Arabic Tweets by Check-worthiness." In CLEF (Working Notes). (2020).
  63. Goudas, Theodosis, Christos Louizos, Georgios Petasis, and Vangelis Karkaletsis. "Argument extraction from news, blogs, and social media." In Hellenic Conference on Artificail Intelli- gence, pp. 287-299. Springer, Cham, (2014).
  64. Sardianos, Christos, Ioannis Manousos Katakis, Georgios Peta- sis, and Vangelis Karkaletsis. "Argument extraction from news." In Proceedings of the 2nd Workshop on Argumentation Mining, pp. 56-66. (2015).
  65. Levy, Ran, Shai Gretz, Benjamin Sznajder, Shay Hummel, Ranit Aharonov, and Noam Slonim. "Unsupervised corpus {wide claim detection." In Proceedings of the 4th Workshop on Argument Mining, pp. 79-84. (2017).
  66. Josue, Caraballo. \PolitiTax A Taxonomy of Political Claims.". (2018). https://https://perma.cc/4RQF-FCPV.
  67. Zarharan, Majid, Samane Ahangar, Fateme Sadat Rezvaninejad, Mahdi Lotfi Bidhendi, Mohammad Taher Pilevar, Behrouz Minaei, and Sauleh Eetemadi. "Persian Stance Classification Data Set." In TTO. 2019.
  68. Samadi, Mohammadreza, Maryam Mousavian, and Saeedeh Momtazi. "Persian fake news detection: Neural representation and classification at word and text levels." Transactions on Asian and Low-Resource Language Information Processing 21, no. 1 (2021): 1-11.
  69. Mottaghi, Vahid, Mahdi Esmaeili, Ghasem Ali Bazaee, and Mo- hammadali Afshar Kazemi. "A decision-making system for de- tecting fake persian news by improving deep learning algo- rithms{case study of Covid-19 news." Journal of applied re- search on industrial engineering 8, no. Special Issue (2021): 1- 17.
  70. Sadr, Mohammad Mohsen, Afshin Mousavi Chelak, Soraya Ziaei, and Jafar Tanha. "A predictive model based on machine learning methods to recognize fake persian news on twitter." International Journal of Nonlinear Analysis and Applications 11 (2020): 119-128.
  71. Sadr, Mohammad Mohsen. "The Use of LSTM Neural Network to Detect Fake News on Persian Twitter." Turkish Journal of Computer and Mathematics Education (TURCOMAT) 12, no. 11 (2021): 6658-6668.
  72. Jahanbakhsh-Nagadeh, Zoleikha, Mohammad-Reza Feizi-De- rakhshi, and Arash Sharifi. "A semi-supervised model for Per- sian rumor verification based on content information." Multi- media Tools and Applications 80, no. 28 (2021): 35267-35295.
  73. Jahanbakhsh-Nagadeh, Zoleikha, Mohammad-Reza Feizi-De- rakhshi, and Arash Sharifi. "A Deep Content-Based Model for Persian Rumor Verification." Transactions on Asian and Low- Resource Language Information Processing 21, no. 1 (2021): 1-29.19
  74. Thoyyibah, Luthfiyatun. "Presupposition Triggers-a Compara- tive Analysis Between Oral News and Written Online News Discourse." JALL (Journal of Applied Linguistics and Liter- acy) 1, no. 2 (2017): 10-23.
  75. Cohen, Jacob. "A coefficient of agreement for nominal scales." Educational and psychological measurement 20, no. 1 (1960): 37-46.
  76. Krippendorff , Klaus. "Validity in content analysis." (1980): 69.
  77. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosu- khin. "Attention is all you need." Advances in neural infor- mation processing systems 30 (2017).
  78. Farahani, Mehrdad, Mohammad Gharachorloo, Marzieh Fara- hani, and Mohammad Manthouri. "Parsbert: Transformer- based model for persian language understanding." Neural Pro- cessing Letters 53, no. 6 (2021): 3831-3847.
  79. Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. "Bert: Pre-training of deep bidirectional transform- ers for language understanding." arXiv preprint arXiv:1810.04805 (2018).
  80. Kingma, Diederik P., and Jimmy Ba. "Adam: A method for sto- chastic optimization." arXiv preprint arXiv:1412.6980 (2014).
  81. Abadi, Martin, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado et al. "Tensorow: Largescale machine learning on heterogeneous distributed sys- tems." arXiv preprint arXiv:1603.04467 (2016).
  82. Tenney, Ian, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim et al. "What do you learn from context? probing for sentence structure in contextu- alized word representations." arXiv preprint arXiv:1905.06316 (2019).
  83. Hewitt, John, and Christopher D. Manning. "A structural probe for finding syntax in word representations". In Proceedings of the 2019 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4129- 4138. (2019).
  84. Ehsan Doostmohammadi, Minoo Nassajian, and Adel Rahimi. "Persian Ezafe Recognition Using Transformers and Its Role in Part-Of-Speech Tagging". In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 961{971, Online. Association for Computational Linguistics. (2020).

Related papers

Check_square at CheckThat! 2020: Claim Detection in Social Media via Fusion of Transformer and Syntactic Features
Sherzod Hakimov

Cornell University - arXiv, 2020

In this digital age of news consumption, a news reader has the ability to react, express and share opinions with others in a highly interactive and fast manner. As a consequence, fake news has made its way into our daily life because of very limited capacity to verify news on the Internet by large companies as well as individuals. In this paper, we focus on solving two problems which are part of the factchecking ecosystem that can help to automate fact-checking of claims in an ever increasing stream of content on social media. For the first problem, claim check-worthiness prediction, we explore the fusion of syntactic features and deep transformer Bidirectional Encoder Representations from Transformers (BERT) embeddings, to classify check-worthiness of a tweet, i.e. whether it includes a claim or not. We conduct a detailed feature analysis and present our best performing models for English and Arabic tweets. For the second problem, claim retrieval, we explore the pre-trained embeddings from a Siamese network transformer model (sentence-transformers) specifically trained for semantic textual similarity, and perform KD-search to retrieve verified claims with respect to a query tweet.

View PDFchevron_right
Check-worthy Claim Detection across Topics for Automated Fact-checking
Amani Abumansour

arXiv (Cornell University), 2022

An important component of an automated fact-checking system is the claim check-worthiness detection system, which ranks sentences by prioritising them based on their need to be checked. Despite a body of research tackling the task, previous research has overlooked the challenging nature of identifying check-worthy claims across different topics. In this paper, we assess and quantify the challenge of detecting check-worthy claims for new, unseen topics. After highlighting the problem, we propose the AraCWA model to mitigate the performance deterioration when detecting check-worthy claims across topics. The AraCWA model enables boosting the performance for new topics by incorporating two components for few-shot learning and data augmentation. Using a publicly available dataset of Arabic tweets consisting of 14 different topics, we demonstrate that our proposed data augmentation strategy achieves substantial improvements across topics overall, where the extent of the improvement varies across topics. Further, we analyse the semantic similarities between topics, suggesting that the similarity metric could be used as a proxy to determine the difficulty level of an unseen topic prior to undertaking the task of labelling the underlying sentences.

View PDFchevron_right
AIT_FHSTP at GermEval 2021: Automatic Fact Claiming Detection with Multilingual Transformer Models
asrun lio

2021

Spreading ones opinion on the internet is becoming more and more important. A problem is that in many discussions people often argue with supposed facts. This year’s GermEval 2021 focuses on this topic by incorporating a shared task on the identification of fact-claiming comments. This paper presents the contribution of the AIT FHSTP team at the GermEval 2021 benchmark for task 3: “identifying fact-claiming comments in social media texts”. Our methodological approaches are based on transformers and incorporate 3 different models: multilingual BERT, GottBERT and XML-RoBERTa. To solve the fact claiming task, we fine-tuned these transformers with external data and the data provided by the GermEval task organizers. Our multilingual BERT model achieved a precision-score of 72.71%, a recall of 72.96% and an F1-Score of 72.84% on the GermEval test set. Our fine-tuned XML-RoBERTa model achieved a precision-score of 68.45%, a recall of 70.11% and a F1-Score of 69.27%. Our best model is GottB...

View PDFchevron_right
Empowering the Fact-checkers! Automatic Identification of Claim Spans on Twitter
Vaibhav Pulastya

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing

The widespread diffusion of medical and political claims in the wake of COVID-19 has led to a voluminous rise in misinformation and fake news. The current vogue is to employ manual fact-checkers to efficiently classify and verify such data to combat this avalanche of claim-ridden misinformation. However, the rate of information dissemination is such that it vastly outpaces the fact-checkers' strength. Therefore, to aid manual fact-checkers in eliminating the superfluous content, it becomes imperative to automatically identify and extract the snippets of claim-worthy (mis)information present in a post. In this work, we introduce the novel task of Claim Span Identification (CSI). We propose CURT, a large-scale Twitter corpus with token-level claim spans on more than 7.5k tweets. Furthermore, along with the standard token classification baselines, we benchmark our dataset with DABERTa, an adapterbased variation of RoBERTa. The experimental results attest that DABERTa outperforms the baseline systems across several evaluation metrics, improving by about 1.5 points. We also report detailed error analysis to validate the model's performance along with the ablation studies. Lastly, we release our comprehensive span annotation guidelines for public use.

View PDFchevron_right
Overview of CheckThat! 2020 English: Automatic Identification and Verification of Claims in Social Media
Reem Suwaileh

2020

We present an overview of the third edition of the CheckThat! Lab at CLEF 2020. The lab featured five tasks in Arabic and English, and here we focus on the three English tasks. Task 1 challenged the participants to predict which tweets from a stream of tweets about COVID-19 are worth fact-checking. Task 2 asked to retrieve verified claims from a set of previously fact-checked claims, which could help fact-check the claims made in an input tweet. Task 5 asked to propose which claims in a political debate or a speech should be prioritized for fact-checking. A total of 18 teams participated in the English tasks, and most submissions managed to achieve sizable improvements over the baselines using models based on BERT, LSTMs, and CNNs. In this paper, we describe the process of data collection and the task setup, including the evaluation measures used, and we give a brief overview of the participating systems. Last but not least, we release to the research community all datasets from the...

View PDFchevron_right

Related topics

  • Multi-Label Classification
  • Low Resource Languages
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved