Academia.eduAcademia.edu

Outline

Title
Abstract
Figures
Introduction
Related Work
Discussion
References

Enriching Word Vectors with Subword Information

Profile image of Peter YuanPeter Yuan

Abstract

Continuous word representations, trained on large unlabeled corpora are useful for many natural language processing tasks. Many popular models to learn such representations ignore the morphology of words, by assigning a distinct vector to each word. This is a limitation , especially for morphologically rich languages with large vocabularies and many rare words. In this paper, we propose a new approach based on the skip-gram model, where each word is represented as a bag of character n-grams. A vector representation is associated to each character n-gram, words being represented as the sum of these representations. Our method is fast, allowing to train models on large corpus quickly. We evaluate the obtained word representations on five different languages, on word similarity and analogy tasks.

Figures (3)
Table 1: Correlations between vector similarity score and human judgement on several datasets (top) and accuracies on analogy tasks (bottom) for models trained on Wikipedia. For each dataset, we evaluate models trained on several sizes of training sets. Small contains 50M tokens, Medium 200M tokens and Full is the complete Wikipedia dump. For each dataset, we report the out-of-vocabulary rate as well as the performance of our model and the skip-gram and CBOW models from/Mikolov et al. (2013b).
Table 1: Correlations between vector similarity score and human judgement on several datasets (top) and accuracies on analogy tasks (bottom) for models trained on Wikipedia. For each dataset, we evaluate models trained on several sizes of training sets. Small contains 50M tokens, Medium 200M tokens and Full is the complete Wikipedia dump. For each dataset, we report the out-of-vocabulary rate as well as the performance of our model and the skip-gram and CBOW models from/Mikolov et al. (2013b).
Table 2: Nearest neighbors of rare words using our representations and skip- gram. These hand picked examples are for illustration.
Table 2: Nearest neighbors of rare words using our representations and skip- gram. These hand picked examples are for illustration.
Table 3: Comparison of our approach with previous work incorporating morphology in word representations, on word similarity tasks. We keep all the word pairs of the evaluation set and obtain representations for out-of-vocabulary words with our model by summing the vectors of character n-grams. We report Spearman’s rank correlation coefficient between model scores and human idaqement
Table 3: Comparison of our approach with previous work incorporating morphology in word representations, on word similarity tasks. We keep all the word pairs of the evaluation set and obtain representations for out-of-vocabulary words with our model by summing the vectors of character n-grams. We report Spearman’s rank correlation coefficient between model scores and human idaqement

Related papers

Enriching Word Embeddings with Global Information and Testing on Highly Inflected Language
Lukáš Svoboda

Computación y Sistemas, 2019

In this paper we evaluate our new approach based on the Continuous Bag-of-Words and Skip-gram models enriched with global context information on highly inflected Czech language and compare it with English results. As a source of information we use Wikipedia, where articles are organized in a hierarchy of categories. These categories provide useful topical information about each article. Both models are evaluated on standard word similarity and word analogy datasets. Proposed models outperform other word representation methods when similar size of training data is used. Model provide similar performance especially with methods trained on much larger datasets.

View PDFchevron_right
Exploiting Morphological Regularities in Distributional Word Representations
arihant gupta

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017

We present a simple, fast and unsupervised approach for exploiting morphological regularities present in high dimensional vector spaces. We propose a novel method for generating embeddings of words from their morphological variants using morphological transformation operators. We evaluate this approach on MSR word analogy test set (Mikolov et al., 2013d) with an accuracy of 85% which is 12% higher than the previous best known system.

View PDFchevron_right
Better Word Embeddings by Disentangling Contextual n-Gram Information
Prakhar Gupta

Proceedings of the 2019 Conference of the North, 2019

Pre-trained word vectors are ubiquitous in Natural Language Processing applications. In this paper, we show how training word embeddings jointly with bigram and even trigram embeddings, results in improved unigram embeddings. We claim that training word embeddings along with higher n-gram embeddings helps in the removal of the contextual information from the unigrams, resulting in better stand-alone word embeddings. We empirically show the validity of our hypothesis by outperforming other competing word representation models by a significant margin on a wide variety of tasks. We make our models publicly available.

View PDFchevron_right
Improving Word Embeddings Using Kernel PCA
Vishwani Gupta

Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019)

Word-based embedding approaches such as Word2Vec capture the meaning of words and relations between them, particularly well when trained with large text collections; however, they fail to do so with small datasets. Extensions such as fastText reduce the amount of data needed slightly, however, the joint task of learning meaningful morphology, syntactic and semantic representations still requires a lot of data. In this paper, we introduce a new approach to warm-start embedding models with morphological information, in order to reduce training time and enhance their performance. We use word embeddings generated using both word2vec and fastText models and enrich them with morphological information of words, derived from kernel principal component analysis (KPCA) of word similarity matrices. This can be seen as explicitly feeding the network morphological similarities and letting it learn semantic and syntactic similarities. Evaluating our models on word similarity and analogy tasks in English and German, we find that they not only achieve higher accuracies than the original skip-gram and fastText models but also require significantly less training data and time. Another benefit of our approach is that it is capable of generating a high-quality representation of infrequent words as, for example, found in very recent news articles with rapidly changing vocabularies. Lastly, we evaluate the different models on a downstream sentence classification task in which a CNN model is initialized with our embeddings and find promising results.

View PDFchevron_right
Distributed representations of words and phrases and their compositionality
Greg Corrado

The recently introduced continuous Skip-gram model is an efficient method for learning high-quality distributed vector representations that capture a large number of precise syntactic and semantic word relationships. In this paper we present several extensions that improve both the quality of the vectors and the training speed. By subsampling of the frequent words we obtain significant speedup and also learn more regular word representations. We also describe a simple alternative to the hierarchical softmax called negative sampling. An inherent limitation of word representations is their indifference to word order and their inability to represent idiomatic phrases. For example, the meanings of "Canada" and "Air" cannot be easily combined to obtain "Air Canada". Motivated by this example, we present a simple method for finding phrases in text, and show that learning good vector representations for millions of phrases is possible.

View PDFchevron_right
hauWE: Hausa Words Embedding for Natural Language Processing
Bashir Galadanci

2019 2nd International Conference of the IEEE Nigeria Computer Chapter (NigeriaComputConf), 2019

Words embedding (distributed word vector representations) have become an essential component of many natural language processing (NLP) tasks such as machine translation, sentiment analysis, word analogy, named entity recognition and word similarity. Despite this, the only work that provides word vectors for Hausa language is that of [1] trained using fastText, consisting of only a few words vectors. This work presents words embedding models using Word2Vec's Continuous Bag of Words (CBoW) and Skip Gram (SG) models. The models, hauWE (Hausa Words Embedding), are bigger and better than the only previous model, making them more useful in NLP tasks. To compare the models, they were used to predict the 10 most similar words to 30 randomly selected Hausa words. hauWE CBoW's 88.7% and hauWE SG's 79.3% prediction accuracy greatly outperformed [1]'s 22.3%.

View PDFchevron_right
Learning Word Embeddings without Context Vectors
Evgenia Elistratova

Most word embedding algorithms such as word2vec or fastText construct two sort of vectors: for words and for contexts. Naive use of vectors of only one sort leads to poor results. We suggest using indefinite inner product in skip-gram negative sampling algorithm. This allows us to use only one sort of vectors without loss of quality. Our "context-free" cf algorithm performs on par with SGNS on word similarity datasets.

View PDFchevron_right
Representation Learning for Unseen Words by Bridging Subwords to Semantic Networks
YeaChan Kim

2020

Pre-trained word embeddings are widely used in various fields. However, the coverage of pre-trained word embeddings only includes words that appeared in corpora where pre-trained embeddings are learned. It means that the words which do not appear in training corpus are ignored in tasks, and it could lead to the limited performance of neural models. In this paper, we propose a simple yet effective method to represent out-of-vocabulary (OOV) words. Unlike prior works that solely utilize subword information or knowledge, our method makes use of both information to represent OOV words. To this end, we propose two stages of representation learning. In the first stage, we learn subword embeddings from the pre-trained word embeddings by using an additive composition function of subwords. In the second stage, we map the learned subwords into semantic networks (e.g., WordNet). We then re-train the subword embeddings by using lexical entries on semantic lexicons that could include newly obser...

View PDFchevron_right
Towards Useful Word Embeddings
Petr Sojka

RASLAN, 2020

Since the seminal work of Mikolov et al. (2013), word vectors of log-bilinear models have found their way into many nlp applications and were extended with the positional model. Although the positional model improves accuracy on the intrinsic English word analogy task, prior work has neglected its evaluation on extrinsic end tasks, which correspond to real-world nlp applications. In this paper, we describe our first steps in evaluating positional weighting on the information retrieval, text classification, and language modeling extrinsic end tasks.

View PDFchevron_right
Adapting Word Embeddings to New Languages with Morphological and Phonological Subword Representations
David R Mortensen

Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018

Much work in Natural Language Processing (NLP) has been for resource-rich languages, making generalization to new, less-resourced languages challenging. We present two approaches for improving generalization to lowresourced languages by adapting continuous word representations using linguistically motivated subword units: phonemes, morphemes and graphemes. Our method requires neither parallel corpora nor bilingual dictionaries and provides a significant gain in performance over previous methods relying on these resources. We demonstrate the effectiveness of our approaches on Named Entity Recognition for four languages, namely Uyghur, Turkish, Bengali and Hindi, of which Uyghur and Bengali are low resource languages, and also perform experiments on Machine Translation. Exploiting subwords with transfer learning gives us a boost of +15.2 NER F1 for Uyghur and +9.7 F1 for Bengali. We also show improvements in the monolingual setting where we achieve (avg.) +3 F1 and (avg.) +1.35 BLEU.

View PDFchevron_right

References (17)

  1. References [Alexandrescu and Kirchhoff2006] Andrei Alexandrescu and Katrin Kirchhoff. 2006. Factored neural language models. In Proc. NAACL. [Ballesteros et al.2015] Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2015. Improved transition-based parsing by modeling characters instead of words with lstms. In Proc. EMNLP. [Baroni and Lenci2010] Marco Baroni and Alessandro Lenci. 2010. Distributional memory: A general framework for corpus-based semantics. Computa- tional Linguistics, 36(4).
  2. Bojanowski et al.2015] Piotr Bojanowski, Armand Joulin, and Tomáš Mikolov. 2015. Alternative structures for character-level rnns. In ICLR Workshop. [Botha and Blunsom2014] Jan A Botha and Phil Blun- som. 2014. Compositional morphology for word rep- resentations and language modelling. In Proc. ICML. [Chen et al.2015] Xinxiong Chen, Lei Xu, Zhiyuan Liu, Maosong Sun, and Huanbo Luan. 2015. Joint learning of character and word embeddings. In Proc. IJCAI.
  3. Grzegorz Chrupała. 2014. Normalizing tweets with edit scripts and recurrent neural embed- dings. In Proc. ACL. [Collobert and Weston2008] Ronan Collobert and Jason Weston. 2008. A unified architecture for natural lan- guage processing: Deep neural networks with multi- task learning. In Proc. ICML. [Cotterell and Schütze2015] Ryan Cotterell and Hinrich Schütze. 2015. Morphological word-embeddings. In Proc. of NAACL. [Cui et al.2015] Qing Cui, Bin Gao, Jiang Bian, Siyu Qiu, Hanjun Dai, and Tie-Yan Liu. 2015. Knet: A general framework for learning word embedding using mor- phological knowledge. ACM Transactions on Infor- mation Systems. [Deerwester et al.1990] Scott Deerwester, Susan T Du- mais, George W Furnas, Thomas K Landauer, and Richard Harshman. 1990. Indexing by latent semantic analysis. Journal of the American society for informa- tion science, 41(6).
  4. dos Santos and Gatti2014] Cicero Nogueira dos Santos and Maira Gatti. 2014. Deep convolutional neural networks for sentiment analysis of short texts. In Proc. COLING. [dos Santos and Zadrozny2014] Cicero Nogueira dos Santos and Bianca Zadrozny. 2014. Learning character-level representations for part-of-speech tagging. In Proc. ICML. [Finkelstein et al.2001] Lev Finkelstein, Evgeniy
  5. Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001. Placing search in context: The concept revisited. In Proc. WWW.
  6. Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850.
  7. Iryna Gurevych. 2005. Using the struc- ture of a conceptual network in computing semantic relatedness. In Proc. IJCNLP.
  8. Zellig S Harris. 1954. Distributional struc- ture. Word, 10(2-3).
  9. Hassan and Mihalcea2009] Samer Hassan and Rada Mi- halcea. 2009. Cross-lingual semantic relatedness us- ing encyclopedic knowledge. In Proc. EMNLP. [Joubarne and Inkpen2011] Colette Joubarne and Diana Inkpen. 2011. Comparison of semantic similarity for different languages using the google n-gram corpus and second-order co-occurrence measures. In Adv. in A.I. [Kim et al.2016] Yoon Kim, Yacine Jernite, David Son- tag, and Alexander M Rush. 2016. Character-aware neural language models. In Proc. AAAI. [Lazaridou et al.2013] Angeliki Lazaridou, Marco
  10. Marelli, Roberto Zamparelli, and Marco Baroni. 2013. Compositional-ly derived representations of morphologically complex words in distributional semantics. In Proc. ACL. [Ling et al.2015] Wang Ling, Chris Dyer, Alan W Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luis Marujo, and Tiago Luis. 2015. Finding function in form: Compositional character models for open vo- cabulary word representation. In Proc. EMNLP. [Lund and Burgess1996] Kevin Lund and Curt Burgess. 1996. Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior Research Meth- ods, Instruments, & Computers, 28(2). [Luong and Manning2016] Minh-Thang Luong and Christopher D. Manning. 2016. Achieving open vocabulary neural machine translation with hybrid word-character models. In Proc. ACL. [Luong et al.2013] Thang Luong, Richard Socher, and Christopher D Manning. 2013. Better word represen- tations with recursive neural networks for morphology. In Proc. CoNLL. [Mikolov et al.2012] Tomáš Mikolov, Ilya Sutskever, Anoop Deoras, Hai-Son Le, Stefan Kom- brink, and J Cernocky. 2012. Subword lan- guage modeling with neural networks. preprint (http://www.fit.vutbr.cz/imikolov/rnnlm/char.pdf). [Mikolov et al.2013a] Tomáš Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estima- tion of word representations in vector space. arXiv preprint arXiv:1301.3781. [Mikolov et al.2013b] Tomáš Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Adv. NIPS. [Qiu et al.2014] Siyu Qiu, Qing Cui, Jiang Bian, Bin Gao, and Tie-Yan Liu. 2014. Co-learning of word represen- tations and morpheme representations. In Proc. COL- ING. [Rumelhart et al.1988] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. 1988. Learning repre- sentations by back-propagating errors. Cognitive mod- eling, 5(3).
  11. Sak et al.2010] Haşim Sak, Murat Saraclar, and Tunga Gungör. 2010. Morphology-based and sub-word lan- guage modeling for turkish speech recognition. In Proc. ICASSP.
  12. Hinrich Schütze. 1992. Dimensions of meaning. In Proc. Supercomputing.
  13. Hinrich Schütze. 1993. Word space. In Adv. NIPS.
  14. Sennrich et al.2016] Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proc. ACL. [Shaoul and Westbury2010] C. Shaoul and C. Westbury. 2010. The Westbury lab Wikipedia corpus. [Soricut and Och2015] Radu Soricut and Franz Och. 2015. Unsupervised morphology induction using word embeddings. In Proc. NAACL.
  15. Charles Spearman. 1904. The proof and measurement of association between two things. American Journal of Psychology, 15. [Sperr et al.2013] Henning Sperr, Jan Niehues, and Alexander Waibel. 2013. Letter n-gram-based in- put encoding for continuous space language models. In Proceedings of the Workshop on Continuous Vector Space Models and their Compositionality. [Sutskever et al.2011] Ilya Sutskever, James Martens, and Geoffrey E Hinton. 2011. Generating text with recur- rent neural networks. In Proc. ICML. [Svoboda and Brychcin2016] L. Svoboda and T. Brychcin. 2016. New word analogy corpus for exploring embeddings of czech words. In Proc. CICLING. [Turney et al.2010] Peter D Turney, Patrick Pantel, et al. 2010. From frequency to meaning: Vector space mod- els of semantics. Journal of artificial intelligence re- search, 37(1).
  16. Zesch and Gurevych2006] Torsten Zesch and Iryna Gurevych. 2006. Automatically creating datasets for measures of semantic relatedness. In Proc. Workshop on Linguistic Distances.
  17. Zhang et al.2015] Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Adv. NIPS.

Related papers

Learning to Generate Word Representations using Subword Information
YeaChan Kim

2018

Distributed representations of words play a major role in the field of natural language processing by encoding semantic and syntactic information of words. However, most existing works on learning word representations typically regard words as individual atomic units and thus are blind to subword information in words. This further gives rise to a difficulty in representing out-of-vocabulary (OOV) words. In this paper, we present a character-based word representation approach to deal with this limitation. The proposed model learns to generate word representations from characters. In our model, we employ a convolutional neural network and a highway network over characters to extract salient features effectively. Unlike previous models that learn word representations from a large corpus, we take a set of pre-trained word embeddings and generalize it to word entries, including OOV words. To demonstrate the efficacy of the proposed model, we perform both an intrinsic and an extrinsic tas...

View PDFchevron_right
Efficient Estimation of Word Representations in Vector Space 2013
丽娜 游

We propose two novel model architectures for computing continuous vector representations of words from very large data sets. The quality of these representations is measured in a word similarity task, and the results are compared to the previously best performing techniques based on different types of neural networks. We observe large improvements in accuracy at much lower computational cost, i.e. it takes less than a day to learn high quality word vectors from a 1.6 billion words data set. Furthermore, we show that these vectors provide state-of-the-art performance on our test set for measuring syntactic and semantic word similarities. arXiv:1301.3781v3 [cs.CL] 7 Sep 2013 1 The test set is available at www.fit.vutbr.cz/˜imikolov/rnnlm/word-test.v1.txt 2

View PDFchevron_right
Efficient Estimation of Word Representations in Vector Space
Greg Corrado

2013

We propose two novel model architectures for computing continuous vector representations of words from very large data sets. The quality of these representations is measured in a word similarity task, and the results are compared to the previously best performing techniques based on different types of neural networks. We observe large improvements in accuracy at much lower computational cost, i.e. it takes less than a day to learn high quality word vectors from a 1.6 billion words data set. Furthermore, we show that these vectors provide state-of-the-art performance on our test set for measuring syntactic and semantic word similarities. arXiv:1301.3781v3 [cs.CL] 7 Sep 2013 1 The test set is available at www.fit.vutbr.cz/˜imikolov/rnnlm/word-test.v1.txt 2

View PDFchevron_right
Better Word Representations with Recursive Neural Networks for Morphology
Christopher D Manning

Vector-space word representations have been very successful in recent years at improving performance across a variety of NLP tasks. However, common to most existing work, words are regarded as independent entities without any explicit relationship among morphologically related words being modeled. As a result, rare and complex words are often poorly estimated, and all unknown words are represented in a rather crude way using only one or a few vectors. This paper addresses this shortcoming by proposing a novel model that is capable of building representations for morphologically complex words from their morphemes. We combine recursive neural networks (RNNs), where each morpheme is a basic unit, with neural language models (NLMs) to consider contextual information in learning morphologicallyaware word representations. Our learned models outperform existing word representations by a good margin on word similarity tasks across many datasets, including a new dataset we introduce focused on rare words to complement existing ones in an interesting way.

View PDFchevron_right
GloVe: Global Vectors for Word Representation
Delia Ioana

Recent methods for learning vector space representations of words have succeeded in capturing fine-grained semantic and syntactic regularities using vector arithmetic , but the origin of these regularities has remained opaque. We analyze and make explicit the model properties needed for such regularities to emerge in word vectors. The result is a new global log-bilinear regression model that combines the advantages of the two major model families in the literature: global matrix factorization and local context window methods. Our model efficiently leverages statistical information by training only on the nonzero elements in a word-word co-occurrence matrix, rather than on the entire sparse matrix or on individual context windows in a large corpus. The model produces a vector space with meaningful sub-structure, as evidenced by its performance of 75% on a recent word analogy task. It also outperforms related models on similarity tasks and named entity recognition.

View PDFchevron_right
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved