Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Experimental Protocol
Results
Conclusions
References
All Topics
Computer Science
Theoretical Computer Science

Unsupervised Statistical Learning of Context-free Grammar

Profile image of Olgierd UnoldOlgierd Unold

2020, Proceedings of the 12th International Conference on Agents and Artificial Intelligence

https://doi.org/10.5220/0009383604310438
visibility

…

description

9 pages

link

1 file

Abstract

In this paper, we address the problem of inducing (weighted) context-free grammar (WCFG) on data given. The induction is performed by using a new model of grammatical inference, i.e., weighted Grammar-based Classifier System (wGCS). wGCS derives from learning classifier systems and searches grammar structure using a genetic algorithm and covering. Weights of rules are estimated by using a novelty Inside-Outside Contrastive Estimation algorithm. The proposed method employs direct negative evidence and learns WCFG both form positive and negative samples. Results of experiments on three synthetic context-free languages show that wGCS is competitive with other statistical-based method for unsupervised CFG learning.

Related papers

Iterative method of generating artificial context-free grammars
Olgierd Unold

2019

Grammatical inference is a machine learning area, whose fundamentals are built around learning sets. At present, real-life data and examples from manually crafted grammars are used to test their learning performance. This paper aims to present a method of generating artificial context-free grammars with their optimal learning sets, which could be successfully applied as a benchmarking tool for empirical grammar inference methods.

View PDFchevron_right
Grammar Induction and Genetic Algorithms-An Overview
nitin choubey

Pacific Journal of Science and …, 2009

Grammar Induction (also know as Grammar Inference or Language Learning) is the process of learning of a grammar from training data. This paper discusses the various approaches for learning context-free grammar (CFG) from the corpus of string and presents the approach of informant learning in the form of result for two standard grammar problems namely Balanced Parenthesis Grammar and Palindrome Grammar.

View PDFchevron_right
Fast Statistical Grammar Induction
Yuji Matsumoto

The statistical induction of context free grammars from bracketed corpora with the Inside Outside Algorithm has often inspired researchers, but the computational complexity has made it impossible to generate a large scale grammar. The method we suggest achieves the same results as earlier research, but at a much smaller expense in computer time. We explain the modifications needed to the algorithm, give results of experiments and compare these to results reported in other literature.

View PDFchevron_right
Guiding unsupervised grammar induction using contrastive estimation
Noah A. Smith

2005

Abstract We describe a novel training criterion for probabilistic grammar induction models, contrastive estimation [Smith and Eisner, 2005], which can be interpreted as exploiting implicit negative evidence and includes a wide class of likelihood-based objective functions. This criterion is a generalization of the function maximized by the Expectation-Maximization algorithm [Dempster et al., 1977]. CE is a natural fit for log-linear models, which can include arbitrary features but for which EM is computationally difficult.

View PDFchevron_right
Genetic Programming for Grammar Induction
Gokturk Ucoluk

2001

There has been a big interest in inducing classes of grammars in the area of machine learning. Various attempts have been carried out for automatically inferring di erent grammar classes. The symbolic nature of the grammar induction problem makes it suitable for the GP-approach. However the straightforward application of the GP method on Context Free Grammar Induction problem fails to generate a satisfactory solution. The interdependency among subparts of a CFG is high and it seems to be the reason that prevents the GP method from nding out e ective building blocks during the search. In this paper a new approach is presented where the aim is to formalize a control module for the genetic search which can use the interdependency information existing in CFGs and hence can direct the search only among well-t grammars in the search space.

View PDFchevron_right
Grammatical inference with grammar-based classifier system
Olgierd Unold

2008

This paper takes up the topic of a task of training Grammar-based Classifier System (GCS) to learn grammar from data. GCS is a new model of Learning Classifier Systems in which the population of classifiers has a form of a context-free grammar rule set in a Chomsky Normal Form. GCS has been proposed to address both the natural language grammar induction and also learning formal grammar for DNA sequence. In both cases near-optimal solutions or better than reported in the literature were obtained.

View PDFchevron_right
A Toolbox for Context-Sensitive Grammar Induction by Genetic Search
Wojciech Wieczorek

2021

This paper introduces a new tool for context-sensitive grammar inference. The source code and library are publicly available via GitHub and NuGet repositories. The article describes the implemented algorithm, input parameters, and the produced output grammar. In addition, the paper contains several use-cases. The described library is written in F#, hence it can be used in any .NET Framework language (F#, C#, C++/CLI, Visual Basic, and J#) and run under the control of varied operating systems.

View PDFchevron_right
Grammar-based Classifier System: A Universal Tool for Grammatical Inference
Olgierd Unold

2008

Grammatical Inference deals with the problem of learning structural models, such as grammars, from different sort of data patterns, such as artificial languages, natural languages, biosequences, speech and so on. This article describes a new grammatical inference tool, Grammar-based Classifier System (GCS) dedicated to learn grammar from data. GCS is a new model of Learning Classifier Systems in which the population of classifiers has a form of a context-free grammar rule set in a Chomsky Normal Form. GCS has been proposed to address both regular language induction and the natural language grammar induction as well as learning formal grammar for DNA sequence. In all cases near-optimal solutions or better than reported in the literature were obtained.

View PDFchevron_right
On unsupervised grammar induction from untagged corpora
Toshikazu Ikuta, Paul Rodrigues

PSiCL: Poznań Studies in Contemporary Linguistics, 2006

In this paper we describe the theoretical motivation and the implementation of an unsupervised morphology induction algorithm. We show that we can induce morphologies with very high precision, using simple unsupervised statistical algorithms. The results are not only relevant from a theoretical point of view, they also have various potential applications in natural language processing.

View PDFchevron_right
Unsupervised Grammar Inference Systems for Natural Language
Eric S Atwell

2002

In recent years there have been significant advances in the field of Unsupervised Grammar Inference (UGI) for Natural Languages such as English or Dutch. This paper presents a broad range of UGI implementations, where we can begin to see how the theory has been put in to practise. Several mature systems are emerging, built using complex models and capable of deriving natural language grammatical phenomena. The range of systems is classified into: models based on Categorial Grammar (GraSp, CLL, EMILE); Memory Based Learning models (FAMBL, RISE); Evolutionary computing models (ILM, LAgts); and string-pattern searches (ABL, GB). An objectively measurable statistical comparison of performance Of the systems reviewed is not yet feasible. However, their merits and shortfalls are discussed, as well as a look at what the future has in store for UGI.

View PDFchevron_right

References (31)

  1. Adriaans, P. and Vervoort, M. (2002). The EMILE 4.1 grammar induction toolbox. In International Col- loquium on Grammatical Inference, pages 293-295. Springer.
  2. Adriaans, P. W. (1992). Language learning from a catego- rial perspective. PhD thesis, Universiteit van Amster- dam.
  3. Baker, J. K. (1979). Trainable grammars for speech recog- nition. The Journal of the Acoustical Society of Amer- ica, 65(S1):S132-S132.
  4. Clark, A. and Lappin, S. (2010). Unsupervised learning and grammar induction. The Handbook of Computational Linguistics and Natural Language Processing, 57.
  5. de la Higuera, C. (2010). Grammatical Inference: Learn- ing Automata and Grammars. Cambridge University Press.
  6. D'Ulizia, A., Ferri, F., and Grifoni, P. (2011). A survey of grammatical inference methods for natural language learning. Artificial Intelligence Review, 36(1):1-27.
  7. Gold, E. M. (1967). Language identification in the limit. Information and control, 10(5):447-474.
  8. Heinz, J., De la Higuera, C., and Van Zaanen, M. (2015). Grammatical inference for computational linguistics. Synthesis Lectures on Human Language Technologies, 8(4):1-139.
  9. Hogenhout, W. R. and Matsumoto, Y. (1998). A fast method for statistical grammar induction. Natural Language Engineering, 4(3):191-209.
  10. Hopcroft, J. E., Motwani, R., and Ullman, J. D. (2001). In- troduction to automata theory, languages, and compu- tation. Acm Sigact News, 32(1):60-65.
  11. Horning, J. J. (1969). A study of grammatical inference. Technical report, Stanford Univ Calif Dept of Com- puter Science.
  12. Johnson, M., Griffiths, T., and Goldwater, S. (2007). Bayesian inference for pcfgs via markov chain monte carlo. In Human Language Technologies 2007: The Conference of the North American Chapter of the As- sociation for Computational Linguistics; Proceedings of the Main Conference, pages 139-146.
  13. Keller, B. and Lutz, R. (1997). Evolving stochastic context- free grammars from examples using a minimum de- scription length principle. In Workshop on Automatic Induction, Grammatical Inference and Language Ac- quisition.
  14. Lari, K. and Young, S. J. (1990). The estimation of stochas- tic context-free grammars using the inside-outside al- gorithm. Computer speech & language, 4(1):35-56.
  15. Nakamura, K. (2003). Incremental learning of context free grammars by extended inductive CYK algorithm. In Proceedings of the 2003rd European Conference on Learning Context-Free Grammars, pages 53-64. Ruder Boskovic Institute.
  16. Petasis, G., Paliouras, G., Karkaletsis, V., Halatsis, C., and Spyropoulos, C. D. (2004). e-GRIDS: Computation- ally efficient gramatical inference from positive exam- ples. Grammars, 7:69-110.
  17. Salkind, N. J. (2010). Encyclopedia of Research Design. SAGE Publications, Inc.
  18. Smith, N. A. and Eisner, J. (2005a). Contrastive estimation: Training log-linear models on unlabeled data. In Pro- ceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pages 354-362. Asso- ciation for Computational Linguistics.
  19. Smith, N. A. and Eisner, J. (2005b). Guiding unsupervised grammar induction using contrastive estimation. In Proc. of IJCAI Workshop on Grammatical Inference Applications, pages 73-82.
  20. Smith, N. A. and Johnson, M. (2007). Weighted and proba- bilistic context-free grammars are equally expressive. Computational Linguistics, 33(4):477-491.
  21. Solan, Z., Horn, D., Ruppin, E., and Edelman, S. (2005). Unsupervised learning of natural languages. Proceedings of the National Academy of Sciences, 102(33):11629-11634.
  22. Stolcke, A. and Omohundro, S. (1994). Inducing prob- abilistic grammars by bayesian model merging. In International Colloquium on Grammatical Inference, pages 106-118. Springer.
  23. Unold, O. (2005). Context-free grammar induction with grammar-based classifier system. Archives of Control Sciences, 15(4):681-690.
  24. Unold, O. (2008). Grammar-based classifier system: a uni- versal tool for grammatical inference. WSEAS Trans- actions on Computers, 7(10):1584-1593.
  25. Unold, O. (2012). Fuzzy grammar-based prediction of amy- loidogenic regions. In International Conference on Grammatical Inference, pages 210-219.
  26. Unold, O. (2019). jGCS. https://github.com/ounold/jGCS.
  27. Unold, O. and Gabor, M. (2019a). How implicit neg- ative evidence improve weighted context-free gram- mar induction. In International Conference on Artifi- cial Intelligence and Soft Computing, pages 595-606. Springer.
  28. Unold, O. and Gabor, M. (2019b). Weighted context-free grammar induction-a preliminary report. In PP- RAI 2019, pages 319-322. ISBN 978-83-943803-2-8.
  29. Urbanowicz, R. J. and Moore, J. H. (2009). Learning clas- sifier systems: a complete introduction, review, and roadmap. Journal of Artificial Evolution and Applica- tions, 2009:1.
  30. Wieczorek, W. (2010). A local search algorithm for gram- matical inference. In Jose M. Sempere, P. G., edi- tor, Grammatical Inference: Theoretical Results and Applications: 10th International Colloquium, ICGI 2010, Valencia, Spain, September 2010. Proceedings, volume 6339 of Lecture Notes in Computer Science, pages 217-229, Berlin, Heidelberg. Springer-Verlag.
  31. Wieczorek, W. and Unold, O. (2016). Use of a novel gram- matical inference approach in classification of amy- loidogenic hexapeptides. Computational and mathe- matical methods in medicine, 2016.

Related papers

Split-Based Algorithm for Weighted Context-Free Grammar Induction
Olgierd Unold

Applied Sciences, 2021

The split-based method in a weighted context-free grammar (WCFG) induction was formalised and verified on a comprehensive set of context-free languages. WCFG is learned using a novel grammatical inference method. The proposed method learns WCFG from both positive and negative samples, whereas the weights of rules are estimated using a novel Inside–Outside Contrastive Estimation algorithm. The results showed that our approach outperforms in terms of F1 scores of other state-of-the-art methods.

View PDFchevron_right
Learning Classifier System Approach to Natural Language Grammar Induction
Olgierd Unold

2007

This paper describes an evolutionary approach to the problem of inferring non-stochastic context-free grammar (CFG) from natural language (NL) corpora. The approach employs Grammar-based Classifier System (GCS). GCS is a new version of Learning Classifier Systems in which classifiers are represented by CFG in Chomsky Normal Form. GCS has been tested on the NL corpora, and it provided comparable results to the pure genetic induction approach, but in a significantly shorter time. The efficient implementation for grammar induction is very important during analysis of large text corpora.

View PDFchevron_right
Context-free grammar induction using evolutionary methods
Olgierd Unold

2003

The research into the ability of building self-learning natural language parser based on context-free grammar (CFG) was presented. The paper investigates the use of evolutionary methods: a genetic algorithm, a genetic programming and learning classifier systems for inferring CFG based parser. The experiments were conducted on the real set of natural language sentences. The gained results confirm the feasibility of applying evolutionary algorithms for context-free grammatical inference.

View PDFchevron_right
Context free grammar induction library using genetic algorithms
Hari Pandey

2010

As we all are aware that "Evolutionary Algorithms" (EAs) are modern techniques used for searching for an optimum. One can establish communication via two medium first oral communications (Speech Processing) and second is written (Text Processing). This project focuses the second medium (text processing) of communication where we communicate using any language by writing some thing. Genetic Algorithms are developed as random search methods, which have not so sensitivity on primary data of the problems. It can be used in estimation of system parameter to get the best possible solution. Genetic Algorithms have been discussed for grammar induction. Grammar Inference or Language Learning is the process of learning grammar from training data.This paper mainly discussed the various methods for learning context-free grammar (CFG) from the corpus of string and presents the approach of informant learning in the form of result for two standard grammar problems first balanced parenthesis problem and second two symbol palindrome over {a, b}.

View PDFchevron_right
A fast method for statistical grammar induction
Yuji Matsumoto

Natural Language Engineering, 1998

The statistical induction of stochastic context free grammars from bracketed corpora with the Inside Outside Algorithm is an appealing method for grammar learning, but the computational complexity of this algorithm has made it impossible to generate a large scale grammar. Researchers from natural language processing and speech recognition have suggested various methods to reduce the computational complexity and, at the same time, guide the learning algorithm towards a solution by, for example, placing constraints on the grammar. We suggest a method that strongly reduces that computational cost of the algorithm without placing constraints on the grammar. This method can in principle be combined with any of the constraints on grammars that have been suggested in earlier studies. We show that it is feasible to achieve results equivalent to earlier research, but with much lower computational effort. After creating a small grammar, the grammar is incrementally increased while rules that ...

View PDFchevron_right

Related topics

  • Computer Science
  • Formal languages
  • Grammar inference
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved