Using suffix arrays as language models: Scaling the n-gram

Proceedings of the 18th conference on Computational linguistics -, 2000

This t)al)t;r de, scribes a hybrid prol)osal to combine n-grams and Stochastic Context-Free Grammars (SCFGs) tbr language modeling. A classical n-gram model is used to cat)lure the local relations between words, while a stochastic grammatical inodel is considered to represent the hmg-term relations between syntactical stru(:tm'es. In order to define this granmlatical model, which will 1)e used on large-vo(:almlary comph'~x tasks, a eategory-t)ased SCFG and a prol)abilisti(" model of' word (tistrilmtion in the categories have been 1)rol)osed. Methods for leanfing these stochastic models tTor complex tasks are described, and algorithms for con> puting the word transition probal)ilities are also 1)resented. Filmily, ext)erilnents using the Penn Treel)ank corpus improved by 30% the test; set; l)erph~xity with regard to the classical n-gram models. * This work has been partially SUl)l)<)rted by the S1)anish CICYT under contract (TIC98/0423-C(16).

Web-scale data has been used in a diverse range of language research. Most of this research has used web counts for only short, fixed spans of con- text. We present a unified view of using web counts for lexical disambiguation. Unlike previous ap- proaches, our supervised and unsupervised systems combine information from multiple and overlap- ping segments of context. On the tasks of preposi- tion selection and context-sensitive spelling correc- tion, the supervised system reduces disambiguation error by 20-24% over the current state-of-the-art.

2008

Accurately translating multiword expressions is important to obtain good performance in machine translation, crosslanguage information retrieval, and other multilingual tasks in human language technology. Existing approaches to inducing translation equivalents of multiword units have focused on agglomerating individual words or on aligning words in a statistical machine translation system. We present a different approach based upon information theoretic heuristics and the exact counting of frequencies of occurrence of multiword strings in aligned parallel corpora. We are applying a technique introduced by Yamamoto and Church that uses suffix arrays and longest common prefix arrays. Evaluation of the method in multiple language pairs was performed using bilingual lexicons of domainspecific terminology as a gold standard. We found that performance of 50-70%, as measured by mean reciprocal rank, can be obtained for terms that occur more than 10 or so times.

Lecture Notes in Computer Science, 2010

Language models (LMs) are essential components of many applications such as speech recognition or machine translation. LMs factorize the probability of a string of words into a product of P(w i |h i ), where h i is the context (history) of word w i . Most LMs use previous words as the context. The paper presents two alternative approaches: post-ngram LMs (which use following words as context) and dependency LMs (which exploit dependency structure of a sentence and can use e.g. the governing word as context). Dependency LMs could be useful whenever a topology of a dependency tree is available, but its lexical labels are unknown, e.g. in tree-to-tree machine translation. In comparison with baseline interpolated trigram LM both of the approaches achieve significantly lower perplexity for all seven tested languages (Arabic, Catalan, Czech, English, Hungarian, Italian, Turkish).

advancedcentrepunjabi.org

N-grams are consecutive overlapping N-character sequences formed from an input stream. N-gram models are extensively used in word sense disambiguation. In this paper we tried to find out whether higher order n gram models improves the word sense disambiguation in Punjabi language and whether it has any relation with entropy of the models. In our experiments statistical analysis of n gram models for n ranging from±1 to±6 is done. We also tried to explore the possibility of disambiguation by using future knowledge. ...

Natural Language Engineering, 2013

This paper presents TrendStream, a versatile architecture for very large word n-gram datasets. Designed for speed, flexibility, and portability, TrendStream uses a novel triebased architecture, features lossless compression, and provides optimization for both speed and memory use. In addition to literal queries, it also supports fast pattern matching searches (with wildcards or regular expressions), on the same data structure, without any additional indexing. Language models are updateable directly in the compiled binary format, allowing rapid encoding of existing tabulated collections, incremental generation of n-gram models from streaming text, and merging of encoded compiled files. This architecture offers flexible choices for loading and memory utilization: fast memory-mapping of a multi-gigabyte model, or on-demand partial data loading with very modest memory requirements. The implemented system runs successfully on several different platforms, under different operating systems, even when the n-gram model file is much larger than available memory. Experimental evaluation results are presented with the Google Web1T collection and the Gigaword corpus.

Proceedings of the Second Workshop on Statistical Machine Translation - StatMT '07, 2007

Statistical machine translation, as well as other areas of human language processing, have recently pushed toward the use of large scale n-gram language models. This paper presents efficient algorithmic and architectural solutions which have been tested within the Moses decoder, an open source toolkit for statistical machine translation. Experiments are reported with a high performing baseline, trained on the Chinese-English NIST 2006 Evaluation task and running on a standard Linux 64-bit PC architecture. Comparative tests show that our representation halves the memory required by SRI LM Toolkit, at the cost of 44% slower translation speed. However, as it can take advantage of memory mapping on disk, the proposed implementation seems to scale-up much better to very large language models: decoding with a 289-million 5-gram language model runs in 2.1Gb of RAM.

Computational Linguistics, 2021

Transformer-based language models have taken many fields in NLP by storm. BERT and its derivatives dominate most of the existing evaluation benchmarks, including those for Word Sense Disambiguation (WSD), thanks to their ability in capturing context-sensitive semantic nuances. However, there is still little knowledge about their capabilities and potential limitations in encoding and recovering word senses. In this article, we provide an in-depth quantitative and qualitative analysis of the celebrated BERT model with respect to lexical ambiguity. One of the main conclusions of our analysis is that BERT can accurately capture high-level sense distinctions, even when a limited number of examples is available for each word sense. Our analysis also reveals that in some cases language models come close to solving coarse-grained noun disambiguation under ideal conditions in terms of availability of training data and computing resources. However, this scenario rarely occurs in real-world se...

2001

In this paper we want to show how Ò-gram language models can be used to provide additional information in automatic speech understanding systems beyond the pure word chain. This becomes important in the context of conversational dialogue systems that have to recognize and interpret spontaneous speech. We show how Ò-grams can (1) help to classify prosodic events like boundaries and accents, (2) be extended to directly provide boundary information in the speech recognition phase, (3) help to process speech repairs, and (4) detect and semantically classify out-of-vocabulary words. The approaches can work on the best word chain or a word hypotheses graph. Examples and experimental results are provided from our own research within the E VAR information retrieval and the VERBMOBIL speech-to-speech translation system.

1999

ABSTRACT The main goal of the present work is to explore the use of rich lexical information in language modelling. We reformulated the task of a language model from predicting the next word given its history to predicting simultaneously both the word and a tag encoding various types of lexical information.