Academia.eduAcademia.edu

Outline

Title
Abstract
Figures
Introduction
Evaluation Methods
Results
Model Analysis: Vector Length and Context Size
Model Analysis: Corpus Size
Model Analysis: Run-Time
Model Analysis: Comparison with WORD2VEC
Conclusion
References

GloVe: Global Vectors for Word Representation

Profile image of Delia IoanaDelia Ioana

Abstract

Recent methods for learning vector space representations of words have succeeded in capturing fine-grained semantic and syntactic regularities using vector arithmetic , but the origin of these regularities has remained opaque. We analyze and make explicit the model properties needed for such regularities to emerge in word vectors. The result is a new global log-bilinear regression model that combines the advantages of the two major model families in the literature: global matrix factorization and local context window methods. Our model efficiently leverages statistical information by training only on the nonzero elements in a word-word co-occurrence matrix, rather than on the entire sparse matrix or on individual context windows in a large corpus. The model produces a vector space with meaningful sub-structure, as evidenced by its performance of 75% on a recent word analogy task. It also outperforms related models on similarity tasks and named entity recognition.

Figures (8)
Table 1: Co-occurrence probabilities for target words ice and steam with selected context words from a 6 billion token corpus. Only in the ratio does noise from non-discriminative words like water and fashion cancel out, so that large values (much greater than 1) correlate well with properties specific to ice, and small values (much less than 1) correlate well with properties specific of steam.
Table 1: Co-occurrence probabilities for target words ice and steam with selected context words from a 6 billion token corpus. Only in the ratio does noise from non-discriminative words like water and fashion cancel out, so that large values (much greater than 1) correlate well with properties specific to ice, and small values (much less than 1) correlate well with properties specific of steam.
Figure 1: Weighting function f with a@ = 3/4. The performance of the model depends weakly on the cutoff, which we fix to Xmax = 100 for all our experiments. We found that a = 3/4 gives a mod- est improvement over a linear version with a = 1. Although we offer only empirical motivation for choosing the value 3/4, it is interesting that a sim- ilar fractional power scaling was found to give the best performance in (Mikolov et al., 2013a).
Figure 1: Weighting function f with a@ = 3/4. The performance of the model depends weakly on the cutoff, which we fix to Xmax = 100 for all our experiments. We found that a = 3/4 gives a mod- est improvement over a linear version with a = 1. Although we offer only empirical motivation for choosing the value 3/4, it is interesting that a sim- ilar fractional power scaling was found to give the best performance in (Mikolov et al., 2013a).
Table 2: Results on the word analogy task, given as percent accuracy. Underlined scores are best within groups of similarly-sized models; bold scores are best overall. HPCA vectors are publicly available”; (i)vLBL results are from (Mnih et al., 2013); skip-gram (SG) and CBOW results are from (Mikolov et al., 2013a,b); we trained SGt and CBOW' using the word2vec tool. See text for details and a description of the SVD models. for details and a description of the SVD models. dataset for NER (Tjong Kim Sang and De Meul- der, 2003). Word analogies. The word analogy task con- sists of questions like, “a is to b as c is The dataset contains 19,544 such questi to?” ons, di- vided into a semantic subset and a syntactic sub- set. The semantic questions are typically analogies about people or places, like “Athens is to Greece as Berlin is to __?”. The syntactic ques ions are typically analogies about verb tenses or forms of adjectives, for example “dance is to dancing as fly is to__?”. To correctly answer the ques ion, the model should uniquely identify the missing term, with only an exact correspondence coun ed as a correct match. We answer the question “a is to b asc isto?” by finding the word d whose repre- sentation wg is closest to Wy — Wg + We according to the cosine similarity.4 Word analogies. The word analogy task con-
Table 2: Results on the word analogy task, given as percent accuracy. Underlined scores are best within groups of similarly-sized models; bold scores are best overall. HPCA vectors are publicly available”; (i)vLBL results are from (Mnih et al., 2013); skip-gram (SG) and CBOW results are from (Mikolov et al., 2013a,b); we trained SGt and CBOW' using the word2vec tool. See text for details and a description of the SVD models. for details and a description of the SVD models. dataset for NER (Tjong Kim Sang and De Meul- der, 2003). Word analogies. The word analogy task con- sists of questions like, “a is to b as c is The dataset contains 19,544 such questi to?” ons, di- vided into a semantic subset and a syntactic sub- set. The semantic questions are typically analogies about people or places, like “Athens is to Greece as Berlin is to __?”. The syntactic ques ions are typically analogies about verb tenses or forms of adjectives, for example “dance is to dancing as fly is to__?”. To correctly answer the ques ion, the model should uniquely identify the missing term, with only an exact correspondence coun ed as a correct match. We answer the question “a is to b asc isto?” by finding the word d whose repre- sentation wg is closest to Wy — Wg + We according to the cosine similarity.4 Word analogies. The word analogy task con-
and differ in that they contain phrase vectors.
and differ in that they contain phrase vectors.
Figure 3: Accuracy on the analogy task for 300- dimensional vectors trained on different corpora.
Figure 3: Accuracy on the analogy task for 300- dimensional vectors trained on different corpora.
shown for neural vectors in (Turian et al., 2010). 4.4 Model Analysis: Vector Length and Context Size In Fig. 2, we show the results of experiments that vary vector length and context window. A context window that extends to the left and right of a tar- get word will be called symmetric, and one which extends only to the left will be called asymmet- ric. In (a), we observe diminishing returns for vec- tors larger than about 200 dimensions. In (b) and (c), we examine the effect of varying the window size for symmetric and asymmetric context win- dows. Performance is better on the syntactic sub- task for small and asymmetric context windows, which aligns with the intuition that syntactic infor- mation is mostly drawn from the immediate con- text and can depend strongly on word order. Se- mantic information, on the other hand, is more fre- quently non-local, and more of it is captured with larger window sizes. In Fig. 2, we show the results of experiments that
shown for neural vectors in (Turian et al., 2010). 4.4 Model Analysis: Vector Length and Context Size In Fig. 2, we show the results of experiments that vary vector length and context window. A context window that extends to the left and right of a tar- get word will be called symmetric, and one which extends only to the left will be called asymmet- ric. In (a), we observe diminishing returns for vec- tors larger than about 200 dimensions. In (b) and (c), we examine the effect of varying the window size for symmetric and asymmetric context win- dows. Performance is better on the syntactic sub- task for small and asymmetric context windows, which aligns with the intuition that syntactic infor- mation is mostly drawn from the immediate con- text and can depend strongly on word order. Se- mantic information, on the other hand, is more fre- quently non-local, and more of it is captured with larger window sizes. In Fig. 2, we show the results of experiments that
it specifies a learning schedule specific to a single pass through the data, making a modification for multiple passes a non-trivial task. Another choice is to vary the number of negative samples. Adding negative samples effectively increases the number of training words seen by the model, so in some ways it is analogous to extra epochs. methods or from prediction-based methods. Cur- rently, prediction-based models garner substantial support; for example, Baroni et al. (2014) argue that these models perform better across a range of tasks. In this work we argue that the two classes of methods are not dramatically different at a fun- damental level since they both probe the under- lying co-occurrence statistics of the corpus, but the efficiency with which the count-based meth- ods capture global statistics can be advantageous. We construct a model that utilizes this main ben- efit of count data while simultaneously capturing the meaningful linear substructures prevalent in recent log-bilinear prediction-based methods like word2vec. The result, GloVe, is a new global log-bilinear regression model for the unsupervised learning of word representations that outperforms other models on word analogy, word similarity, and named entity recognition tasks.
it specifies a learning schedule specific to a single pass through the data, making a modification for multiple passes a non-trivial task. Another choice is to vary the number of negative samples. Adding negative samples effectively increases the number of training words seen by the model, so in some ways it is analogous to extra epochs. methods or from prediction-based methods. Cur- rently, prediction-based models garner substantial support; for example, Baroni et al. (2014) argue that these models perform better across a range of tasks. In this work we argue that the two classes of methods are not dramatically different at a fun- damental level since they both probe the under- lying co-occurrence statistics of the corpus, but the efficiency with which the count-based meth- ods capture global statistics can be advantageous. We construct a model that utilizes this main ben- efit of count data while simultaneously capturing the meaningful linear substructures prevalent in recent log-bilinear prediction-based methods like word2vec. The result, GloVe, is a new global log-bilinear regression model for the unsupervised learning of word representations that outperforms other models on word analogy, word similarity, and named entity recognition tasks.

Related papers

Matrix Factorization using Window Sampling and Negative Sampling for Improved Word Representations
Marco idiart

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2016

In this paper, we propose LexVec, a new method for generating distributed word representations that uses low-rank, weighted factorization of the Positive Point-wise Mutual Information matrix via stochastic gradient descent, employing a weighting scheme that assigns heavier penalties for errors on frequent co-occurrences while still accounting for negative co-occurrence. Evaluation on word similarity and analogy tasks shows that LexVec matches and often outperforms state-of-the-art methods on many of these tasks. * This is a preprint of the paper that will be presented at the 54th Annual Meeting of the Association for Computational Linguistics.

View PDFchevron_right
Embedding Semantic Relations into Word Representations
Ken-ichi Kawarabayashi

Learning representations for semantic relations is important for various tasks such as analogy detection , relational search, and relation classification. Although there have been several proposals for learning representations for individual words, learning word representations that explicitly capture the semantic relations between words remains under developed. We propose an unsupervised method for learning vector representations for words such that the learnt representations are sensitive to the semantic relations that exist between two words. First, we extract lexical patterns from the co-occurrence contexts of two words in a corpus to represent the semantic relations that exist between those two words. Second, we represent a lexical pattern as the weighted sum of the representations of the words that co-occur with that lexical pattern. Third, we train a binary classifier to detect relationally similar vs. non-similar lexical pattern pairs. The proposed method is unsupervised in ...

View PDFchevron_right
Word Embedding Evaluation in Downstream Tasks and Semantic Analogies
Bernardo Consoli

2020

Language Models have long been a prolific area of study in the field of Natural Language Processing (NLP). One of the newer kinds of language models, and some of the most used, are Word Embeddings (WE). WE are vector space representations of a vocabulary learned by a non-supervised neural network based on the context in which words appear. WE have been widely used in downstream tasks in many areas of study in NLP. These areas usually use these vector models as a feature in the processing of textual data. This paper presents the evaluation of newly released WE models for the Portuguese langauage, trained with a corpus composed of 4.9 billion tokens. The first evaluation presented an intrinsic task in which WEs had to correctly build semantic and syntactic relations. The second evaluation presented an extrinsic task in which the WE models were used in two downstream tasks: Named Entity Recognition and Semantic Similarity between Sentences. Our results show that a diverse and comprehen...

View PDFchevron_right
Improving Word Embeddings Using Kernel PCA
Vishwani Gupta

Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019)

Word-based embedding approaches such as Word2Vec capture the meaning of words and relations between them, particularly well when trained with large text collections; however, they fail to do so with small datasets. Extensions such as fastText reduce the amount of data needed slightly, however, the joint task of learning meaningful morphology, syntactic and semantic representations still requires a lot of data. In this paper, we introduce a new approach to warm-start embedding models with morphological information, in order to reduce training time and enhance their performance. We use word embeddings generated using both word2vec and fastText models and enrich them with morphological information of words, derived from kernel principal component analysis (KPCA) of word similarity matrices. This can be seen as explicitly feeding the network morphological similarities and letting it learn semantic and syntactic similarities. Evaluating our models on word similarity and analogy tasks in English and German, we find that they not only achieve higher accuracies than the original skip-gram and fastText models but also require significantly less training data and time. Another benefit of our approach is that it is capable of generating a high-quality representation of infrequent words as, for example, found in very recent news articles with rapidly changing vocabularies. Lastly, we evaluate the different models on a downstream sentence classification task in which a CNN model is initialized with our embeddings and find promising results.

View PDFchevron_right
Contextualizing Semantic Representations Using Syntactically Enriched Vector Models
Stefan Thater

Meeting of the Association for Computational Linguistics, 2010

We present a syntactically enriched vector model that supports the computation of contextualized semantic representations in a quasi compositional fashion. It employs a systematic combination of first-and second-order context vectors. We apply our model to two different tasks and show that (i) it substantially outperforms previous work on a paraphrase ranking task, and (ii) achieves promising results on a wordsense similarity task; to our knowledge, it is the first time that an unsupervised method has been applied to this task.

View PDFchevron_right
Joint Word Representation Learning Using a Corpus and a Semantic Lexicon
Ken-ichi Kawarabayashi

Proceedings of the AAAI Conference on Artificial Intelligence

Methods for learning word representations using large text corpora have received much attention lately due to their impressive performancein numerous natural language processing (NLP) tasks such as, semantic similarity measurement, and word analogy detection.Despite their success, these data-driven word representation learning methods do not considerthe rich semantic relational structure between words in a co-occurring context. On the other hand, already much manual effort has gone into the construction of semantic lexicons such as the WordNetthat represent the meanings of words by defining the various relationships that exist among the words in a language.We consider the question, can we improve the word representations learnt using a corpora by integrating theknowledge from semantic lexicons?. For this purpose, we propose a joint word representation learning method that simultaneously predictsthe co-occurrences of two words in a sentence subject to the relational constrains given ...

View PDFchevron_right
Understanding the Source of Semantic Regularities in Word Embeddings
Zachary Pardos

Proceedings of the 24th Conference on Computational Natural Language Learning

Semantic relations are core to how humans understand and express concepts in the real world using language. Recently, there has been a thread of research aimed at modeling these relations by learning vector representations from text corpora. Most of these approaches focus strictly on leveraging the co-occurrences of relationship word pairs within sentences. In this paper, we investigate the hypothesis that examples of a lexical relation in a corpus are fundamental to a neural word embedding's ability to complete analogies involving the relation. Our experiments, in which we remove all known examples of a relation from training corpora, show only marginal degradation in analogy completion performance involving the removed relation. This finding enhances our understanding of neural word embeddings, showing that co-occurrence information of a particular semantic relation is the not the main source of their structural regularity.

View PDFchevron_right
Combining Word Representations for Measuring Word Relatedness and Similarity
Dipesh Gautam

2015

Many unsupervised methods, such as Latent Semantic Analysis and Latent Dirichlet Allocation, have been proposed to automatically infer word representations in the form of a vector. By representing a word by a vector, one can exploit the power of vector algebra to solve many Natural Language Processing tasks e.g. by computing the cosine similarity between the corresponding word vectors the semantic similarity between the two words can be captured. In this paper, we hypothesize that combining different word representations complements the coverage of semantic aspects of a word and thus better represents the word than the individual representations. To this end, we present two approaches of combining word representations obtained from many heterogeneous sources. We also report empirical results for word-to-word semantic similarity and relatedness by using the new representation using two existing benchmark datasets.

View PDFchevron_right
Intrinsic analysis for dual word embedding space models
MOHIT MAYANK (RA2111004010364)

arXiv (Cornell University), 2020

Recent word embeddings techniques represent words in a continuous vector space, moving away from the atomic and sparse representations of the past. Each such technique can further create multiple varieties of embeddings based on different settings of hyper-parameters like embedding dimension size, context window size and training method. One additional variety appears when we especially consider the Dual embedding space techniques which generate not one but two-word embeddings as output. This gives rise to an interesting question-"is there one or a combination of the two word embeddings variety, which works better for a specific task?". This paper tries to answer this question by considering all of these variations. Herein, we compare two classical embedding methods belonging to two different methodologies-Word2Vec from window-based and Glove from count-based. For an extensive evaluation after considering all variations, a total of 84 different models were compared against semantic, association and analogy evaluations tasks which are made up of 9 open-source linguistics datasets. The final Word2vec reports showcase the preference of non-default model for 2 out of 3 tasks. In case of Glove, non-default models outperform in all 3 evaluation tasks.

View PDFchevron_right
Word2Vec: Optimal hyperparameters and their impact on natural language processing downstream tasks
Foteini Liwicki

Open Computer Science, 2022

Word2Vec is a prominent model for natural language processing tasks. Similar inspiration is found in distributed embeddings (word-vectors) in recent state-of-the-art deep neural networks. However, wrong combination of hyperparameters can produce embeddings with poor quality. The objective of this work is to empirically show that Word2Vec optimal combination of hyper-parameters exists and evaluate various combinations. We compare them with the publicly released, original Word2Vec embedding. Both intrinsic and extrinsic (downstream) evaluations are carried out, including named entity recognition and sentiment analysis. Our main contributions include showing that the best model is usually task-specific, high analogy scores do not necessarily correlate positively with F1 scores, and performance is not dependent on data size alone. If ethical considerations to save time, energy, and the environment are made, then relatively smaller corpora may do just as well or even better in some cases...

View PDFchevron_right

References (29)

  1. Tom M. Apostol. 1976. Introduction to Analytic Number Theory. Introduction to Analytic Num- ber Theory.
  2. Marco Baroni, Georgiana Dinu, and Germán Kruszewski. 2014. Don't count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors. In ACL.
  3. Yoshua Bengio. 2009. Learning deep architectures for AI. Foundations and Trends in Machine Learning.
  4. Yoshua Bengio, Réjean Ducharme, Pascal Vin- cent, and Christian Janvin. 2003. A neural prob- abilistic language model. JMLR, 3:1137-1155.
  5. John A. Bullinaria and Joseph P. Levy. 2007. Ex- tracting semantic representations from word co- occurrence statistics: A computational study. Behavior Research Methods, 39(3):510-526.
  6. Dan C. Ciresan, Alessandro Giusti, Luca M. Gam- bardella, and Jürgen Schmidhuber. 2012. Deep neural networks segment neuronal membranes in electron microscopy images. In NIPS, pages 2852-2860.
  7. Ronan Collobert and Jason Weston. 2008. A uni- fied architecture for natural language process- ing: deep neural networks with multitask learn- ing. In Proceedings of ICML, pages 160-167.
  8. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural Language Processing (Al- most) from Scratch. JMLR, 12:2493-2537.
  9. Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41.
  10. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learn- ing and stochastic optimization. JMLR, 12.
  11. Lev Finkelstein, Evgenly Gabrilovich, Yossi Ma- tias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001. Placing search in con- text: The concept revisited. In Proceedings of the 10th international conference on World Wide Web, pages 406-414. ACM.
  12. Eric H. Huang, Richard Socher, Christopher D. Manning, and Andrew Y. Ng. 2012. Improving Word Representations via Global Context and Multiple Word Prototypes. In ACL.
  13. Rémi Lebret and Ronan Collobert. 2014. Word embeddings through Hellinger PCA. In EACL.
  14. Omer Levy, Yoav Goldberg, and Israel Ramat- Gan. 2014. Linguistic regularities in sparse and explicit word representations. CoNLL-2014.
  15. Kevin Lund and Curt Burgess. 1996. Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior Research Methods, In- strumentation, and Computers, 28:203-208.
  16. Minh-Thang Luong, Richard Socher, and Christo- pher D Manning. 2013. Better word represen- tations with recursive neural networks for mor- phology. CoNLL-2013.
  17. Tomas Mikolov, Kai Chen, Greg Corrado, and Jef- frey Dean. 2013a. Efficient Estimation of Word Representations in Vector Space. In ICLR Work- shop Papers.
  18. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. In NIPS, pages 3111-3119.
  19. Tomas Mikolov, Wen tau Yih, and Geoffrey Zweig. 2013c. Linguistic regularities in con- tinuous space word representations. In HLT- NAACL.
  20. George A. Miller and Walter G. Charles. 1991. Contextual correlates of semantic similarity. Language and cognitive processes, 6(1):1-28.
  21. Andriy Mnih and Koray Kavukcuoglu. 2013. Learning word embeddings efficiently with noise-contrastive estimation. In NIPS.
  22. Douglas L. T. Rohde, Laura M. Gonnerman, and David C. Plaut. 2006. An improved model of semantic similarity based on lexical co-occurence. Communications of the ACM, 8:627-633.
  23. Herbert Rubenstein and John B. Goodenough. 1965. Contextual correlates of synonymy. Com- munications of the ACM, 8(10):627-633.
  24. Fabrizio Sebastiani. 2002. Machine learning in au- tomated text categorization. ACM Computing Surveys, 34:1-47.
  25. Richard Socher, John Bauer, Christopher D. Man- ning, and Andrew Y. Ng. 2013. Parsing With Compositional Vector Grammars. In ACL.
  26. Stefanie Tellex, Boris Katz, Jimmy Lin, Aaron Fernandes, and Gregory Marton. 2003. Quanti- tative evaluation of passage retrieval algorithms for question answering. In Proceedings of the SIGIR Conference on Research and Develop- ment in Informaion Retrieval.
  27. Erik F. Tjong Kim Sang and Fien De Meul- der. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named en- tity recognition. In CoNLL-2003.
  28. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and gen- eral method for semi-supervised learning. In Proceedings of ACL, pages 384-394.
  29. Mengqiu Wang and Christopher D. Manning. 2013. Effect of non-linear deep architecture in sequence labeling. In Proceedings of the 6th International Joint Conference on Natural Lan- guage Processing (IJCNLP).

Related papers

Efficient Estimation of Word Representations in Vector Space
Greg Corrado

2013

We propose two novel model architectures for computing continuous vector representations of words from very large data sets. The quality of these representations is measured in a word similarity task, and the results are compared to the previously best performing techniques based on different types of neural networks. We observe large improvements in accuracy at much lower computational cost, i.e. it takes less than a day to learn high quality word vectors from a 1.6 billion words data set. Furthermore, we show that these vectors provide state-of-the-art performance on our test set for measuring syntactic and semantic word similarities. arXiv:1301.3781v3 [cs.CL] 7 Sep 2013 1 The test set is available at www.fit.vutbr.cz/˜imikolov/rnnlm/word-test.v1.txt 2

View PDFchevron_right
Efficient Estimation of Word Representations in Vector Space 2013
丽娜 游

We propose two novel model architectures for computing continuous vector representations of words from very large data sets. The quality of these representations is measured in a word similarity task, and the results are compared to the previously best performing techniques based on different types of neural networks. We observe large improvements in accuracy at much lower computational cost, i.e. it takes less than a day to learn high quality word vectors from a 1.6 billion words data set. Furthermore, we show that these vectors provide state-of-the-art performance on our test set for measuring syntactic and semantic word similarities. arXiv:1301.3781v3 [cs.CL] 7 Sep 2013 1 The test set is available at www.fit.vutbr.cz/˜imikolov/rnnlm/word-test.v1.txt 2

View PDFchevron_right
Improving Word Representations via Global Context and Multiple Word Prototypes
Christopher D Manning

Unsupervised word representations are very useful in NLP tasks both as inputs to learning algorithms and as extra word features in NLP systems. However, most of these models are built with only local context and one representation per word. This is problematic because words are often polysemous and global context can also provide useful information for learning word meanings. We present a new neural network architecture which 1) learns word embeddings that better capture the semantics of words by incorporating both local and global document context, and 2) accounts for homonymy and polysemy by learning multiple embeddings per word. We introduce a new dataset with human judgments on pairs of words in sentential context, and evaluate our model on it, showing that our model outperforms competitive baselines and other neural language models. 1

View PDFchevron_right
Towards Useful Word Embeddings
Petr Sojka

RASLAN, 2020

Since the seminal work of Mikolov et al. (2013), word vectors of log-bilinear models have found their way into many nlp applications and were extended with the positional model. Although the positional model improves accuracy on the intrinsic English word analogy task, prior work has neglected its evaluation on extrinsic end tasks, which correspond to real-world nlp applications. In this paper, we describe our first steps in evaluating positional weighting on the information retrieval, text classification, and language modeling extrinsic end tasks.

View PDFchevron_right
Enriching Word Vectors with Subword Information
Peter Yuan

Continuous word representations, trained on large unlabeled corpora are useful for many natural language processing tasks. Many popular models to learn such representations ignore the morphology of words, by assigning a distinct vector to each word. This is a limitation , especially for morphologically rich languages with large vocabularies and many rare words. In this paper, we propose a new approach based on the skip-gram model, where each word is represented as a bag of character n-grams. A vector representation is associated to each character n-gram, words being represented as the sum of these representations. Our method is fast, allowing to train models on large corpus quickly. We evaluate the obtained word representations on five different languages, on word similarity and analogy tasks.

View PDFchevron_right
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved