A Hybrid Approach for Multiword Expression Identification

International Journal of Engineering & Technology, 2018

A three phase hybrid method for automatic extraction of English multiword expressions (MWEs) has been proposed. The proposed method is based on linguistic patterns, association and context similarity between constituent words of the MWEs. First, the expressions are extract-ed in the form of N-grams from the raw text and then filtered using well defined linguistic patterns. Next, these expressions are again fil-tered using association score and context similarity score between their constituent words. Two association measures, Dice’s coefficient and PMI have been used for calculating the association score. The context similarity between words has been calculated using Latent Semantic Analysis (LSA) method. The problem of deciding the best value for the cut-off boundary thresholds in statistical methods is quite common. A two phase method of deciding the boundary threshold, using training dataset, has been proposed and employed in the current work. De-tailed performance analysis has b...

Cet article présente la méthodologie suivie et les résultats obtenus dans le cadre d'un projet qui a pour objectif la construction d'une large base de données d'expressions multi-mots de la langue portugaise. Ces expressions multi-mots ont été automatiquement extraites d'un corpus équilibré de 50 millions de mots, interprétées statistiquement à l'aide de mesures d'association lexicales et ont été ensuite manuellement vérifiées. La base de données lexicales recouvre différent types d'expressions multi-mots avec différents degrés de cohésion, qui vont de la quasi totale fixité jusqu'aux groupes de mots qui se réalisent préférentiellement ensemble, comme les collocations. Le large ensemble de données de cette ressource permettra une révision des typologies d'unités multi-mots en portugais et l'évaluation de différentes mesures d'associations lexicales.

Language Resources and Evaluation, 2010

Due to idiosyncrasies in their syntax, semantics or frequency, Multiword Expressions (MWEs) have received special attention from the NLP community, as the methods and techniques developed for the treatment of simplex words are not necessarily suitable for them. This is certainly the case for the automatic acquisition of MWEs from corpora. A lot of effort has been directed to the task of automatically identifying them, with considerable success. In this paper, we propose an approach for the identification of MWEs in a multilingual context, as a by-product of a word alignment process, that not only deals with the identification of possible MWE candidates, but also associates some multiword expressions with semantics. The results obtained indicate the feasibility and low costs in terms of tools and resources demanded by this approach, which could, for example, facilitate and speed up lexicographic work.

Proceedings of the Workshop on Multiword Expressions Identification, Interpretation, Disambiguation and Applications - MWE '09, 2009

Multiword Expressions (MWEs) are one of the stumbling blocks for more precise Natural Language Processing (NLP) systems. Particularly, the lack of coverage of MWEs in resources can impact negatively on the performance of tasks and applications, and can lead to loss of information or communication errors. This is especially problematic in technical domains, where a significant portion of the vocabulary is composed of MWEs. This paper investigates the use of a statisticallydriven alignment-based approach to the identification of MWEs in technical corpora. We look at the use of several sources of data, including parallel corpora, using English and Portuguese data from a corpus of Pediatrics, and examining how a second language can provide relevant cues for this tasks. We report results obtained by a combination of statistical measures and linguistic information, and compare these to the reported in the literature. Such an approach to the (semi-)automatic identification of MWEs can considerably speed up lexicographic work, providing a more targeted list of MWE candidates.

ArXiv, 2016

The project aims to provide a semi-supervised approach to identify Multiword Expressions in a multilingual context consisting of English and most of the major Indian languages. Multiword expressions are a group of words which refers to some conventional or regional way of saying things. If they are literally translated from one language to another the expression will lose its inherent meaning. To automatically extract multiword expressions from a corpus, an extraction pipeline have been constructed which consist of a combination of rule based and statistical approaches. There are several types of multiword expressions which differ from each other widely by construction. We employ different methods to detect different types of multiword expressions. Given a POS tagged corpus in English or any Indian language the system initially applies some regular expression filters to narrow down the search space to certain patterns (like, reduplication, partial reduplication, compound nouns, comp...

2014

A. Introduction Definition This paper presents a first approach to the development of an algorithm that would automatically detect multiword expression (MWE). For the moment, our work focuses on nominal multiword expressions, since encoding their features according to grammar rules seems a rather feasible task. A nominal multiword expression or a compound noun is a sequence of words that function as a noun, forming one lexical item. Most of the times, the meaning of the whole does not derive from the meaning of the parts. For example, παιδική χαρά [lit. “kids’ joy”, meaning “playground”], ψήφος εμπιστοσύνης [“vote of confidence”],

Proceedings of the Workshop on a Broader Perspective on Multiword Expressions - MWE '07, 2007

Natural languages contain many multi-word sequences that do not display the variety of syntactic processes we would expect given their phrase type, and consequently must be included in the lexicon as multiword units. This paper describes a method for identifying such items in corpora, focussing on English verb-noun combinations. In an evaluation using a set of dictionary-published MWEs we show that our method achieves greater accuracy than existing MWE extraction methods based on lexical association.

This paper explores the full-fledged supervised Machine Learning based approach for the automatic extraction of lexical chunks, commonly called as Multi-Word Expression (MWE). The concept of MWE concerns a variety of constructions in everyday language in the form of idioms, phrasal verbs and noun compounds. The pervasiveness of MWE in the NLP tasks that deals with real text, such as Machine Translation and Information retrieval should be provided with enough MWE treatment; if not, the system will fail to generate high-quality natural output. Here, we are extracting phrasal verbs from the English movie subtitle corpus based on their corresponding linguistic pattern and standard association scores. The extracted phrasal verbs have been used to train various machine learning algorithms for discriminating MWE. Two methods of linguistic pattern extraction are implemented, out of which one is proven to be effective. Here, we have demonstrated two major findings, 1) MWE extraction based on dependency information along with POS tag provides better accuracy than it had been extracted from the POS tag pattern alone. 2) The result obtained from extraction is used to train three different machine learning classifiers, out of which Random forest classifier is verified to be the suitable classifier for the application handled.

Computational Linguistics, 2017

Multiword expressions (MWEs) are a class of linguistic forms spanning conventional word boundaries that are both idiosyncratic and pervasive across different languages. The structure of linguistic processing that depends on the clear distinction between words and phrases has to be re-thought to accommodate MWEs. The issue of MWE handling is crucial for NLP applications, where it raises a number of challenges. The emergence of solutions in the absence of guiding principles motivates this survey, whose aim is not only to provide a focused review of MWE processing, but also to clarify the nature of interactions between MWE processing and downstream applications. We propose a conceptual framework within which challenges and research contributions can be positioned. It offers a shared understanding of what is meant by “MWE processing,” distinguishing the subtasks of MWE discovery and identification. It also elucidates the interactions between MWE processing and two use cases: Parsing and...

2014

Although corpus size is a well known factor that affects the performance of many NLP tasks, for many languages large freely available corpora are still scarce. In this paper we describe one effort to build a very large corpus for Brazilian Portuguese, the brWaC, generated following the Web as Corpus kool yinitiative. To indirectly assess the quality of the resulting corpus we examined the impact of corpus origin in a specific task, the identification of Multiword Expressions with association measures, against a standard corpus. Focusing on nominal compounds, the expressions obtained from each corpus are of comparable quality and indicate that corpus origin has no impact on this task.