Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Experimental Setup
Probability Fraction Results
Discussion
References
All Topics
Computer Science
Natural Language Processing

Automating Text Naturalness Evaluation of NLG Systems

Profile image of E CE C

2020

Abstract

Automatic methods and metrics that assess various quality criteria of automatically generated texts are important for developing NLG systems because they produce repeatable results and allow for a fast development cycle. We present here an attempt to automate the evaluation of text naturalness which is a very important characteristic of natural language generation methods. Instead of relying on human participants for scoring or labeling the text samples, we propose to automate the process by using a human likeliness metric we define and a discrimination procedure based on large pretrained language models with their probability distributions. We analyze the text probability fractions and observe how they are influenced by the size of the generative and discriminative models involved in the process. Based on our results, bigger generators and larger pretrained discriminators are more appropriate for a better evaluation of text naturalness. A comprehensive validation procedure with human participants is required as follow up to check how well this automatic evaluation scheme correlates with human judgments.

Related papers

Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)
Philippe Dessus

2021

Sentence-level text simplification is evaluated using both automated metrics and human evaluation. For automatic evaluation, a combination of metrics is usually employed to evaluate different aspects of the simplification. Flesch-Kincaid Grade Level (FKGL) is one metric that has been regularly used to measure the readability of system output. In this paper, we argue that FKGL should not be used to evaluate text simplification systems. We provide experimental analyses on recent system output showing that the FKGL score can easily be manipulated to improve the score dramatically with only minor impact on other automated metrics (BLEU and SARI). Instead of using FKGL, we suggest that the component statistics, along with others, be used for posthoc analysis to understand system behavior. Percent Modified 0 100 Approach none randomperiod random-the replacelongest replace-randperiod replace-randthe rand-period+ repl-longest BLEU 1-gram 0.4590 0.4300 0.4394 0.

View PDFchevron_right
Can we evaluate the quality of generated text?
Donia Scott

The 6th edition of the Language Resources and …, 2008

Evaluating the output of NLG systems is notoriously difficult, and performing assessments of text quality even more so. A range of automated and subject-based approaches to the evaluation of text quality have been taken, including comparison with a putative gold standard text, analysis of specific linguistic features of the output, expert review and task-based evaluation. In this paper we present the results of a variety of such approaches in the context of a case study application. We discuss the problems encountered in the implementation of each approach in the context of the literature, and propose that a test based on the Turing test for machine intelligence offers a way forward in the evaluation of the subjective notion of text quality.

View PDFchevron_right
Towards evaluation in natural language generation
Robert Dale

1998

What role should evaluation play in the development of natural language generation (NLG) techniques and systems? In this paper we begin by characterising the NLG task, and in the light of this characterisation we discuss how the problems of NLG evaluation are different from the problems of evaluating natural language understanding. We then suggest a way forward for evaluation of work in natural language generation that proceeds by looking more closely at the component problems that are addressed in research in the field.

View PDFchevron_right
A Brief Survey on Natural Language Processing Based Text Generation and Evaluation Techniques
Sidra Minhas

2022

Text Generation is a pressing topic of Natural Language Processing that involves the prediction of upcoming text. Applications like auto-complete, chatbots, auto-correct, and many others use text generation to meet certain communicative requirements. However more accurate text generation methods are needed to encapsulate all possibilities of natural language communication. In this survey, we present cutting-edge methods being adopted for text generation. These methods are divided into three broad categories i.e. 1) Sequence-to-Sequence models (Seq2Seq), 2) Generative Adversarial Networks (GAN), and 3) Miscellaneous. Sequence-to-Sequence involves supervised methods, while GANs are unsupervised, aimed at reducing the dependence of models on training data. After this, we also list a few other text generation methods. We also summarize some evaluation metrics available for text generation and their Performance

View PDFchevron_right
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation
E. Krahmer, Albert Gatt

This paper surveys the current state of the art in Natural Language Generation (nlg), defined as the task of generating text or speech from non-linguistic input. A survey of nlg is timely in view of the changes that the field has undergone over the past decade or so, especially in relation to new (usually data-driven) methods, as well as new applications of nlg technology. This survey therefore aims to (a) give an up-to-date synthesis of research on the core tasks in nlg and the architectures adopted in which such tasks are organised; (b) highlight a number of relatively recent research topics that have arisen partly as a result of growing synergies between nlg and other areas of artificial intelligence; (c) draw attention to the challenges in nlg evaluation, relating them to similar challenges faced in other areas of nlp, with an emphasis on different evaluation methods and the relationships between them.

View PDFchevron_right
Natural Language Generation The State of the Art and the Concept of Ph
Jakub Sido

2020

Computational systems use natural language for communication with humans more often in the last years. This work summarises state-of-the-art approaches in the field of generative models, especially in the text domain. It offers a complex study of specific problems known from this domain and related ones like adversarial training, reinforcement learning, artificial neural networks, etc. It also addresses the usage of these models in the context of non-generative approaches and the possibility of combining both. This work was supported by Grant No. SGS-2019-018 Processing of heterogeneous data and its specialized applications. Copies of this report are available on http://www.kiv.zcu.cz/en/research/publications/ or by surface mail on request sent to the following address: University of West Bohemia Department of Computer Science and Engineering Univerzitní 8 30614 Plzeň Czech Republic Copyright c ○ 2020 University of West Bohemia, Czech Republic

View PDFchevron_right
Text content and task performance in the evaluation of a natural language generation system
Albert Gatt

2009

Abstract An important question in the evaluation of Natural Language Generation systems concerns the relationship between textual characteristics and task performance. If the results of task-based evaluation can be correlated to properties of the text, there are better prospects for improving the system.

View PDFchevron_right
Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation
Mingkai Deng

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021

Natural language generation (NLG) spans a broad range of tasks, each of which serves for specific objectives and desires different properties of generated text. The complexity makes automatic evaluation of NLG particularly challenging. Previous work has typically focused on a single task and developed individual evaluation metrics based on specific intuitions. In this paper, we propose a unifying perspective that facilitates the design of metrics for a wide range of language generation tasks and quality aspects. Based on the nature of information change from input to output, we classify NLG tasks into compression (e.g., summarization), transduction (e.g., text rewriting), and creation (e.g., dialog). The information alignment, or overlap, between input, context, and output text plays a common central role in characterizing the generation. Using the uniform concept of information alignment, we develop a family of interpretable metrics for various NLG tasks and aspects, often without need of gold reference data. To operationalize the metrics, we train self-supervised models to approximate information alignment as a prediction task. Experiments show the uniformly designed metrics achieve stronger or comparable correlations with human judgement compared to state-of-the-art metrics in each of diverse tasks, including text summarization, style transfer, and knowledge-grounded dialog. With information alignment as the intermediate representation, we deliver a composable library for easy NLG evaluation and future metric design. 1

View PDFchevron_right
Perturbation CheckLists for Evaluating NLG Evaluation Metrics
Tanay Dixit

ArXiv, 2021

Natural Language Generation (NLG) evaluation is a multifaceted task requiring assessment of multiple desirable criteria, e.g., fluency, coherency, coverage, relevance, adequacy, overall quality, etc. Across existing datasets for 6 NLG tasks, we observe that the human evaluation scores on these multiple criteria are often not correlated. For example, there is a very low correlation between human scores on fluency and data coverage for the task of structured data to text generation. This suggests that the current recipe of proposing new automatic evaluation metrics for NLG by showing that they correlate well with scores assigned by humans for a single criteria (overall quality) alone is inadequate. Indeed, our extensive study involving 25 automatic evaluation metrics across 6 different tasks and 18 different evaluation criteria shows that there is no single metric which correlates well with human scores on all desirable criteria, for most NLG tasks. Given this situation, we propose Ch...

View PDFchevron_right
Natural Language Generation
Michael Zock

Oxford Handbooks Online, 2005

Communication via a natural language requires two fundamental skills, producing 'text' (written or spoken) and understanding it. Th is chapter introduces newcomers to the fi eld of computational approaches to the former-natural language generation (henceforth NLG)-showing some of the theoretical and practical problems that linguists, computer scientists, and psychologists have encountered when trying to explain how language works in machines or in our minds. ¹ Authors in alphabetical order     . G I: W  N L G (C, L,  S D)

View PDFchevron_right

References (49)

  1. Aguilar-Ruiz, J., Bacardit, J., Divina, F.: Experimental evaluation of discretization schemes for rule induction. In: Deb, K. (ed.) Genetic and Evolutionary Computa- tion -GECCO 2004. pp. 828-839. Springer Berlin Heidelberg (2004)
  2. Ambati, B.R., Reddy, S., Steedman, M.: Assessing relative sentence com- plexity using an incremental CCG parser. In: Proceedings of the 2016 Con- ference of the North American Chapter of the Association for Compu- tational Linguistics. pp. 1051-1057. Association for Computational Linguis- tics, San Diego, California (Jun 2016). https://doi.org/10.18653/v1/N16-1120, https://www.aclweb.org/anthology/N16-1120
  3. Amidei, J., Piwek, P., Willis, A.: Evaluation methodologies in auto- matic question generation 2013-2018. In: Proceedings of the 11th In- ternational Conference on Natural Language Generation. pp. 307- 317. Association for Computational Linguistics, Tilburg University, The Netherlands (Nov 2018). https://doi.org/10.18653/v1/W18-6537, https://www.aclweb.org/anthology/W18-6537
  4. Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learn- ing to align and translate. In: 3rd International Conference on Learning Repre- sentations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (2015), http://arxiv.org/abs/1409.0473
  5. Banerjee, S., Lavie, A.: METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In: Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. pp. 65-72. Association for Computational Linguistics, Ann Arbor, Michigan (Jun 2005), https://www.aclweb.org/anthology/W05-0909
  6. Beltagy, I., Lo, K., Cohan, A.: SciBERT: A pretrained language model for scientific text. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 3615-3620. Association for Computational Lin- guistics, Hong Kong, China (Nov 2019). https://doi.org/10.18653/v1/D19-1371, https://www.aclweb.org/anthology/D19-1371
  7. Bojar, O., Kos, K., Mareček, D.: Tackling sparse data issue in machine transla- tion evaluation. In: Proceedings of the ACL 2010 Conference Short Papers. pp. 86-91. Association for Computational Linguistics, Uppsala, Sweden (Jul 2010), https://www.aclweb.org/anthology/P10-2016
  8. C ¸ano, E., Bojar, O.: Efficiency metrics for data-driven models: A text sum- marization case study. In: Proceedings of the 12th International Conference on Natural Language Generation. pp. 229-239. Association for Computational Lin- guistics, Tokyo, Japan (Oct-Nov 2019). https://doi.org/10.18653/v1/W19-8630, https://www.aclweb.org/anthology/W19-8630
  9. C ¸ano, E., Bojar, O.: Keyphrase generation: A multi-aspect survey. In: 2019 25th Conference of Open Innovations Association (FRUCT). pp. 85-94. Helsinki, Finland (Nov 2019). https://doi.org/10.23919/FRUCT48121.2019.8981519, https://ieeexplore.ieee.org/document/8981519
  10. C ¸ano, E., Bojar, O.: Human or machine: Automating human likeliness evaluation of nlg texts. CoRR abs/2006.03189 (2020), https://arxiv.org/abs/2006.03189
  11. C ¸ano, E., Bojar, O.: Two huge title and keyword generation corpora of research articles. In: Proceedings of The 12th Language Resources and Evaluation Confer- ence. pp. 6663-6671. European Language Resources Association, Marseille, France (may 2020), https://www.aclweb.org/anthology/2020.lrec-1.823
  12. C ¸ano, E., Bojar, O.: Sentiment analysis of czech texts: An algorithmic survey. In: Proceedings of the 11th International Conference on Agents and Artificial Intelligence -Volume 2: NLPinAI,. pp. 973-979. INSTICC, SciTePress (2019). https://doi.org/10.5220/0007695709730979
  13. C ¸ano, E., Morisio, M.: A data-driven neural network archi- tecture for sentiment analysis. Data Technologies and Applica- tions 53(1), 2-19 (2019). https://doi.org/10.1108/DTA-03-2018-0017, https://doi.org/10.1108/DTA-03-2018-0017
  14. Chen, P., Wu, F., Wang, T., Ding, W.: A semantic qa-based approach for text summarization evaluation. In: McIlraith, S.A., Weinberger, K.Q. (eds.) Proceed- ings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018. pp. 4800-4807. AAAI Press (2018), https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16115
  15. Denkowski, M., Lavie, A.: Choosing the Right Evaluation for Machine Transla- tion: an Examination of Annotator and Automatic Metric Performance on Human Judgment Tasks. In: Proceedings of AMTA 2010 (2010)
  16. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pp. 4171-4186. Association for Computational Linguis- tics, Minneapolis, Minnesota (Jun 2019). https://doi.org/10.18653/v1/N19-1423, https://www.aclweb.org/anthology/N19-1423
  17. Fornaciari, T., Poesio, M.: Identifying fake Amazon reviews as learn- ing from crowds. In: Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguis- tics. pp. 279-287. Association for Computational Linguistics, Gothen- burg, Sweden (Apr 2014). https://doi.org/10.3115/v1/E14-1030, https://www.aclweb.org/anthology/E14-1030
  18. Gehrmann, S., Strobelt, H., Rush, A.: GLTR: Statistical detection and visualization of generated text. In: Proceedings of the 57th An- nual Meeting of the Association for Computational Linguistics: System Demonstrations. pp. 111-116. Association for Computational Linguis- tics, Florence, Italy (Jul 2019). https://doi.org/10.18653/v1/P19-3019, https://www.aclweb.org/anthology/P19-3019
  19. Gkatzia, D., Mahamood, S.: A snapshot of NLG evaluation practices 2005 -2014. In: Proceedings of the 15th European Workshop on Natural Lan- guage Generation (ENLG). pp. 57-60. Association for Computational Lin- guistics, Brighton, UK (Sep 2015). https://doi.org/10.18653/v1/W15-4708, https://www.aclweb.org/anthology/W15-4708
  20. Hastie, H., Belz, A.: A comparative evaluation methodology for nlg in interac- tive systems. In: Proceedings of LREC'14. pp. 4004-4011. European Language Resources Association (12 2014)
  21. Kim, H., Lim, J.H., Kim, H.K., Na, S.H.: Qe bert: Bilingual bert using multi- task learning for neural quality estimation. In: Proceedings of the Fourth Con- ference on Machine Translation (Volume 3: Shared Task Papers, Day 2). pp. 87-91. Association for Computational Linguistics, Florence, Italy (August 2019), http://www.aclweb.org/anthology/W19-5407
  22. Klein, G., Kim, Y., Deng, Y., Senellart, J., Rush, A.: OpenNMT: Open-source toolkit for neural machine translation. In: Proceedings of ACL 2017, System Demonstrations. pp. 67-72. Association for Computational Linguistics, Vancou- ver, Canada (Jul 2017), https://www.aclweb.org/anthology/P17-4012
  23. van der Lee, C., Gatt, A., van Miltenburg, E., Wubben, S., Krah- mer, E.: Best practices for the human evaluation of automatically gener- ated text. In: Proceedings of the 12th International Conference on Natu- ral Language Generation. pp. 355-368. Association for Computational Lin- guistics, Tokyo, Japan (Oct-Nov 2019). https://doi.org/10.18653/v1/W19-8643, https://www.aclweb.org/anthology/W19-8643
  24. Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C.H., Kang, J.: BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36(4), 1234-1240 (09 2019). https://doi.org/10.1093/bioinformatics/btz682
  25. Lin, C.Y.: Rouge: A package for automatic evaluation of summaries. In: Proc. ACL workshop on Text Summarization Branches Out. p. 10 (2004), http://aclweb.org/anthology/W04-1013
  26. Liu, Y., Lapata, M.: Text summarization with pretrained encoders. In: Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP). pp. 3730-3740. Association for Computational Lin- guistics, Hong Kong, China (Nov 2019). https://doi.org/10.18653/v1/D19-1387, https://www.aclweb.org/anthology/D19-1387
  27. Maas, A.L., Daly, R.E., Pham, P.T., Huang, D., Ng, A.Y., Potts, C.: Learning word vectors for sentiment analysis. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. pp. 142-150. Association for Computational Linguistics, Portland, Oregon, USA (June 2011), http://www.aclweb.org/anthology/P11-1015
  28. Malmasi, S., Zampieri, M.: Detecting hate speech in social media. In: Pro- ceedings of the Recent Advances in Natural Language Processing Con- ference (RANLP 2017). p. 467-472. Varna, Bulgaria (September 2017). https://doi.org/10.26615/978-954-452-049-6 062
  29. Mozafari, M., Farahbakhsh, R., Crespi, N.: A bert-based transfer learning approach for hate speech detection in online social media. In: Cherifi, H., Gaito, S., Mendes, J.F., Moro, E., Rocha, L.M. (eds.) Complex Networks and Their Applications VIII. pp. 928-940. Springer International Publishing, Cham (2020)
  30. Nallapati, R., Zhou, B., dos Santos, C., Gulcehre, C., Xiang, B.: Ab- stractive text summarization using sequence-to-sequence rnns and be- yond. In: Proceedings of The 20th SIGNLL Conference on Computa- tional Natural Language Learning. pp. 280-290. Association for Com- putational Linguistics (2016). https://doi.org/10.18653/v1/K16-1028, http://aclweb.org/anthology/K16-1028
  31. Novikova, J., Dušek, O., Cercas Curry, A., Rieser, V.: Why we need new evaluation metrics for NLG. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. pp. 2241-2252. Association for Computational Linguistics, Copen- hagen, Denmark (Sep 2017). https://doi.org/10.18653/v1/D17-1238, https://www.aclweb.org/anthology/D17-1238
  32. Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for au- tomatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (2002), http://aclweb.org/anthology/P02-1040
  33. Pérez-Rosas, V., Kleinberg, B., Lefevre, A., Mihalcea, R.: Automatic detection of fake news. In: Proceedings of the 27th International Conference on Computational Linguistics. pp. 3391-3401. Association for Computational Linguistics, Santa Fe, New Mexico, USA (Aug 2018), https://www.aclweb.org/anthology/C18-1287
  34. Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., Zettlemoyer, L.: Deep contextualized word representations. In: Proceedings of the 2018 Conference of the North American Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). pp. 2227-2237. Association for Computational Linguis- tics, New Orleans, Louisiana (Jun 2018). https://doi.org/10.18653/v1/N18-1202, https://www.aclweb.org/anthology/N18-1202
  35. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners (2018), https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf
  36. Reiter, E.: A structured review of the validity of BLEU. Computational Linguistics 44(3), 393-401 (Sep 2018). https://doi.org/10.1162/coli a 00322, https://www.aclweb.org/anthology/J18-3002
  37. Reiter, E., Belz, A.: An investigation into the validity of some metrics for automatically evaluating natural language generation systems. Computational Linguistics 35(4), 529-558 (2009). https://doi.org/10.1162/coli.2009.35.4.35405, https://www.aclweb.org/anthology/J09-4008
  38. Shu, K., Sliva, A., Wang, S., Tang, J., Liu, H.: Fake news detec- tion on social media: A data mining perspective. SIGKDD Explor. Newsl. 19(1), 2236 (Sep 2017). https://doi.org/10.1145/3137597.3137600, https://doi.org/10.1145/3137597.3137600
  39. Sulem, E., Abend, O., Rappoport, A.: BLEU is not suitable for the evaluation of text simplification. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pp. 738-744. Association for Computational Linguis- tics, Brussels, Belgium (Oct-Nov 2018). https://doi.org/10.18653/v1/D18-1081, https://www.aclweb.org/anthology/D18-1081
  40. Sun, C., Myers, A., Vondrick, C., Murphy, K., Schmid, C.: Videobert: A joint model for video and language representation learning. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 7463-7472. Seoul, South Korea (2019)
  41. Tu, Z., Lu, Z., Liu, Y., Liu, X., Li, H.: Modeling coverage for neu- ral machine translation. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. pp. 76-85. Association for Computational Linguistics (2016). https://doi.org/10.18653/v1/P16-1008, http://aclweb.org/anthology/P16-1008
  42. Vajjala, S., Meurers, D.: Assessing the relative reading level of sentence pairs for text simplification. In: Proceedings of the 14th Conference of the Eu- ropean Chapter of the Association for Computational Linguistics (EACL- 14). Association for Computational Linguistics, Gothenburg, Sweden (2014), http://purl.org/dm/papers/Vajjala.Meurers-14-eacl.html
  43. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L.u., Polosukhin, I.: Attention is all you need. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems 30, pp. 5998-6008. Curran Associates, Inc. (2017), http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf
  44. Wang, A., Cho, K.: BERT has a mouth, and it must speak: BERT as a Markov random field language model. In: Proceedings of the Work- shop on Methods for Optimizing and Evaluating Neural Language Gen- eration. pp. 30-36. Association for Computational Linguistics, Min- neapolis, Minnesota (Jun 2019). https://doi.org/10.18653/v1/W19-2304, https://www.aclweb.org/anthology/W19-2304
  45. Young, T., Hazarika, D., Poria, S., Cambria, E.: Recent trends in deep learning based natural language processing [review article].
  46. IEEE Computational Intelligence Magazine 13(3), 55-75 (Aug 2018). https://doi.org/10.1109/MCI.2018.2840738
  47. Zellers, R., Holtzman, A., Rashkin, H., Bisk, Y., Farhadi, A., Roesner, F., Choi, Y.: Defending against neural fake news. In: Advances in Neural Infor- mation Processing Systems 32, pp. 9054-9065. Curran Associates, Inc. (2019), http://papers.nips.cc/paper/9106-defending-against-neural-fake-news.pdf
  48. Zhang, Z., Robinson, D., Tepper, J.: Detecting hate speech on twitter using a convolution-gru based deep neural network. In: Gangemi, A., Navigli, R., Vidal, M.E., Hitzler, P., Troncy, R., Hollink, L., Tordai, A., Alam, M. (eds.) The Semantic Web. pp. 745-760. Springer International Publishing, Cham (2018)
  49. Zhou, W., Xu, K.: Learning to compare for better training and evaluation of open domain natural language generation models. In: The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innova- tive Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020. pp. 9717-9724. AAAI Press (2020), https://aaai.org/ojs/index.php/AAAI/article/view/6521

Related papers

Human or Machine: Automating Human Likeliness Evaluation of NLG Texts
E C

2020

Automatic evaluation of various text quality criteria produced by data-driven intelligent methods is very common and useful because it is cheap, fast, and usually yields repeatable results. In this paper, we present an attempt to automate the human likeliness evaluation of the output text samples coming from natural language generation methods used to solve several tasks. We propose to use a human likeliness score that shows the percentage of the output samples from a method that look as if they were written by a human. Instead of having human participants label or rate those samples, we completely automate the process by using a discrimination procedure based on large pretrained language models and their probability distributions. As follow up, we plan to perform an empirical analysis of human-written and machine-generated texts to find the optimal setup of this evaluation approach. A validation procedure involving human participants will also check how the automatic evaluation correlates with human judgments.

View PDFchevron_right
RoMe: A Robust Metric for Evaluating Natural Language Generation
Md Rashad Al Hasan Rony

arXiv (Cornell University), 2022

Evaluating Natural Language Generation (NLG) systems is a challenging task. Firstly, the metric should ensure that the generated hypothesis reflects the reference's semantics. Secondly, it should consider the grammatical quality of the generated sentence. Thirdly, it should be robust enough to handle various surface forms of the generated sentence. Thus, an effective evaluation metric has to be multifaceted. In this paper, we propose an automatic evaluation metric incorporating several core aspects of natural language understanding (language competence, syntactic and semantic variation). Our proposed metric, RoMe, is trained on language features such as semantic similarity combined with tree edit distance and grammatical acceptability, using a self-supervised neural network to assess the overall quality of the generated sentence. Moreover, we perform an extensive robustness analysis of the state-of-the-art methods and RoMe. Empirical results suggest that RoMe has a stronger correlation to human judgment over state-of-the-art metrics in evaluating system-generated sentences across several NLG tasks.

View PDFchevron_right
RankME: Reliable Human Ratings for Natural Language Generation
Jekaterina Novikova

Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), 2018

Human evaluation for natural language generation (NLG) often suffers from inconsistent user ratings. While previous research tends to attribute this problem to individual user preferences, we show that the quality of human judgements can also be improved by experimental design. We present a novel rank-based magnitude estimation method (RankME), which combines the use of continuous scales and relative assessments. We show that RankME significantly improves the reliability and consistency of human ratings compared to traditional evaluation methods. In addition, we show that it is possible to evaluate NLG systems according to multiple, distinct criteria, which is important for error analysis. Finally, we demonstrate that RankME, in combination with Bayesian estimation of system quality, is a cost-effective alternative for ranking multiple NLG systems.

View PDFchevron_right
Text Generation and Enhanced Evaluation of Metric for Machine Translation
Lata Ragha

2021

View PDFchevron_right
A Survey of Text Generation Models
Vrushali Kulkarni

International Journal of Computer Applications

Natural language processing (NLP)is a field of linguistics and computer science which focuses on the interaction of humans and computers. The main aim of natural language processing is to make sure that a computer can understand what a human says and possibly get key insights from the auditory data. Natural language text production is a well-known sub-part of NLP which focuses on converting auditory data from spoken languages into text. This survey aims to shed some light on crucial details about the past, present and the future of text production algorithms along with an aim to provide a comprehensive overview of how different machine learning techniques are being investigated and studied for different NLP applications. Finally, some important research gaps which were found out through the review are highlighted as the study is drawn to a close. This study also aims to synthesize a guide for beginners in this field and to point them towards related research and popular practices.

View PDFchevron_right

Related topics

  • Information Retrieval
  • Natural Language Processing
  • Natural Language Generation
  • Computational Linguistics
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved