Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Background
Methodology
Analysis
WORD2VEC Analysis
Fasttext Analysis
Glove Analysis
Summary Recommendations
Conclusion
References

Task-dependent Optimal Weight Combinations for Static Embeddings

Profile image of Nathaniel CarlsonNathaniel Carlson

Northern European Journal of Language Technology

https://doi.org/10.3384/NEJLT.2000-1533.2022.4438
visibility

…

description

15 pages

link

1 file

Abstract

A variety of NLP applications use word2vec skip-gram, GloVe, and fastText word embeddings. These models learn two sets of embedding vectors, but most practitioners use only one of them, or alternately an unweighted sum of both. This is the first study to systematically explore a range of linear combinations between the first and second embedding sets. We evaluate these combinations on a set of six NLP benchmarks including IR, POS-tagging, and sentence similarity. We show that the default embedding combinations are often suboptimal and demonstrate 1.0-8.0% improvements. Notably, GloVe’s default unweighted sum is its least effective combination across tasks. We provide a theoretical basis for weighting one set of embeddings more than the other according to the algorithm and task. We apply our findings to improve accuracy in applications of cross-lingual alignment and navigational knowledge by up to 15.2%.

Related papers

A Comparison of Architectures and Pretraining Methods for Contextualized Multilingual Word Embeddings
Niels van der Heijden

Proceedings of the AAAI Conference on Artificial Intelligence

The lack of annotated data in many languages is a well-known challenge within the field of multilingual natural language processing (NLP). Therefore, many recent studies focus on zero-shot transfer learning and joint training across languages to overcome data scarcity for low-resource languages. In this work we (i) perform a comprehensive comparison of state-of-the-art multilingual word and sentence encoders on the tasks of named entity recognition (NER) and part of speech (POS) tagging; and (ii) propose a new method for creating multilingual contextualized word embeddings, compare it to multiple baselines and show that it performs at or above state-of-the-art level in zero-shot transfer settings. Finally, we show that our method allows for better knowledge sharing across languages in a joint training setting.

View PDFchevron_right
GloVe: Global Vectors for Word Representation
Delia Ioana

Recent methods for learning vector space representations of words have succeeded in capturing fine-grained semantic and syntactic regularities using vector arithmetic , but the origin of these regularities has remained opaque. We analyze and make explicit the model properties needed for such regularities to emerge in word vectors. The result is a new global log-bilinear regression model that combines the advantages of the two major model families in the literature: global matrix factorization and local context window methods. Our model efficiently leverages statistical information by training only on the nonzero elements in a word-word co-occurrence matrix, rather than on the entire sparse matrix or on individual context windows in a large corpus. The model produces a vector space with meaningful sub-structure, as evidenced by its performance of 75% on a recent word analogy task. It also outperforms related models on similarity tasks and named entity recognition.

View PDFchevron_right
Obtaining Better Static Word Embeddings Using Contextual Embedding Models
Prakhar Gupta

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021

The advent of contextual word embeddingsrepresentations of words which incorporate semantic and syntactic information from their context-has led to tremendous improvements on a wide variety of NLP tasks. However, recent contextual models have prohibitively high computational cost in many use-cases and are often hard to interpret. In this work, we demonstrate that our proposed distillation method, which is a simple extension of CBOW-based training, allows to significantly improve computational efficiency of NLP applications, while outperforming the quality of existing static embeddings trained from scratch as well as those distilled from previously proposed methods. As a side-effect, our approach also allows a fair comparison of both contextual and static embeddings via standard lexical evaluation tasks.

View PDFchevron_right
How to evaluate word embeddings? On importance of data efficiency and simple supervised tasks
Stanislaw Jastrzebski

2017

Maybe the single most important goal of representation learning is making subsequent learning faster. Surprisingly, this fact is not well reflected in the way embeddings are evaluated. In addition, recent practice in word embeddings points towards importance of learning specialized representations. We argue that focus of word representation evaluation should reflect those trends and shift towards evaluating what useful information is easily accessible. Specifically, we propose that evaluation should focus on data efficiency and simple supervised tasks, where the amount of available data is varied and scores of a supervised model are reported for each subset (as commonly done in transfer learning). In order to illustrate significance of such analysis, a comprehensive evaluation of selected word embeddings is presented. Proposed approach yields a more complete picture and brings new insight into performance characteristics, for instance information about word similarity or analogy ten...

View PDFchevron_right
Word Embedding Evaluation in Downstream Tasks and Semantic Analogies
Bernardo Consoli

2020

Language Models have long been a prolific area of study in the field of Natural Language Processing (NLP). One of the newer kinds of language models, and some of the most used, are Word Embeddings (WE). WE are vector space representations of a vocabulary learned by a non-supervised neural network based on the context in which words appear. WE have been widely used in downstream tasks in many areas of study in NLP. These areas usually use these vector models as a feature in the processing of textual data. This paper presents the evaluation of newly released WE models for the Portuguese langauage, trained with a corpus composed of 4.9 billion tokens. The first evaluation presented an intrinsic task in which WEs had to correctly build semantic and syntactic relations. The second evaluation presented an extrinsic task in which the WE models were used in two downstream tasks: Named Entity Recognition and Semantic Similarity between Sentences. Our results show that a diverse and comprehen...

View PDFchevron_right
Wan2vec: Embeddings learned on word association norms
Gerardo Sierra

Semantic Web

Word embeddings are powerful for many tasks in natural language processing. In this work, we learn word embeddings using weighted graphs from word association norms (WAN) with the node2vec algorithm. Although building WAN is a difficult and time-consuming task, training the vectors from these resources is a fast and efficient process. This allows us to obtain good quality word embeddings from small corpora. We evaluate our word vectors in two ways: intrinsic and extrinsic. The intrinsic evaluation was performed with several word similarity benchmarks, WordSim-353, MC30, MTurk-287, MEN-TR-3k, SimLex-999, MTurk-771 and RG-65, and different similarity measures achieving better results than those obtained with word2vec, GloVe, and fastText, trained on a huge corpus. The extrinsic evaluation was done by measuring the quality of sentence embeddings using transfer tasks: sentiment analysis, paraphrase detection, natural language inference, and semantic textual similarity.

View PDFchevron_right
The Limitations of Cross-language Word Embeddings Evaluation
Roman Suvorov

Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, 2018

The aim of this work is to explore the possible limitations of existing methods of crosslanguage word embeddings evaluation, addressing the lack of correlation between intrinsic and extrinsic cross-language evaluation methods. To prove this hypothesis, we construct English-Russian datasets for extrinsic and intrinsic evaluation tasks and compare performances of 5 different cross-language models on them. The results say that the scores even on different intrinsic benchmarks do not correlate to each other. We can conclude that the use of human references as ground truth for cross-language word embeddings is not proper unless one does not understand how do native speakers process semantics in their cognition.

View PDFchevron_right
Efficient Estimation of Word Representations in Vector Space 2013
丽娜 游

We propose two novel model architectures for computing continuous vector representations of words from very large data sets. The quality of these representations is measured in a word similarity task, and the results are compared to the previously best performing techniques based on different types of neural networks. We observe large improvements in accuracy at much lower computational cost, i.e. it takes less than a day to learn high quality word vectors from a 1.6 billion words data set. Furthermore, we show that these vectors provide state-of-the-art performance on our test set for measuring syntactic and semantic word similarities. arXiv:1301.3781v3 [cs.CL] 7 Sep 2013 1 The test set is available at www.fit.vutbr.cz/˜imikolov/rnnlm/word-test.v1.txt 2

View PDFchevron_right
Learning Word Embeddings without Context Vectors
Evgenia Elistratova

Most word embedding algorithms such as word2vec or fastText construct two sort of vectors: for words and for contexts. Naive use of vectors of only one sort leads to poor results. We suggest using indefinite inner product in skip-gram negative sampling algorithm. This allows us to use only one sort of vectors without loss of quality. Our "context-free" cf algorithm performs on par with SGNS on word similarity datasets.

View PDFchevron_right
Efficient Estimation of Word Representations in Vector Space
Greg Corrado

2013

We propose two novel model architectures for computing continuous vector representations of words from very large data sets. The quality of these representations is measured in a word similarity task, and the results are compared to the previously best performing techniques based on different types of neural networks. We observe large improvements in accuracy at much lower computational cost, i.e. it takes less than a day to learn high quality word vectors from a 1.6 billion words data set. Furthermore, we show that these vectors provide state-of-the-art performance on our test set for measuring syntactic and semantic word similarities. arXiv:1301.3781v3 [cs.CL] 7 Sep 2013 1 The test set is available at www.fit.vutbr.cz/˜imikolov/rnnlm/word-test.v1.txt 2

View PDFchevron_right

References (53)

  1. Akhtar, Syed Sarfaraz, Arihant Gupta, Avijit Vajpayee, Arjit Srivastava, and Manish Shrivastava. 2017. Word similarity datasets for Indian languages: Annotation and baseline systems. In Proceedings of the 11th Lin- guistic Annotation Workshop, pages 91-94, Valencia, Spain. Association for Computational Linguistics.
  2. Andrus, Berkeley and Nancy Fulda. 2020. Immer- sive gameplay via improved natural language under- standing. In Foundations of Digital Games 2020.
  3. Attardi, Giusepppe. 2015. Wikiextractor. https://github. com/attardi/wikiextractor.
  4. Bandy, John and Nicholas Vincent. 2021. Addressing "documentation debt" in machine learning: A retro- spective datasheet for bookcorpus. In Proceedings Northern European Journal of Language Technology of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1.
  5. Bender, Emily M., Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dan- gers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, page 610-623, New York, NY, USA. Association for Computing Machinery.
  6. Bojanowski, Piotr, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
  7. Brown, Zachary, Nathaniel Robinson, David Wingate, and Nancy Fulda. 2020. Towards neural program- ming interfaces. Advances in Neural Information Pro- cessing Systems, 33:17416-17428.
  8. Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computa- tional Linguistics.
  9. Dufter, Philipp, Nora Kassner, and Hinrich Schütze. 2021. Static embeddings as efficient knowledge bases? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, pages 2353-2363.
  10. Fulda, Nancy, Daniel Ricks, Ben Murdoch, and David Wingate. 2017a. What can you do with a rock? affor- dance extraction via word embeddings. In Proceed- ings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pages 1039-1045.
  11. Fulda, Nancy and Nathaniel Robinson. 2021. Improved word representations via summed target and context embeddings. In 2020 IEEE 18th World Symposium on Applied Machine Intelligence and Informatics (SAMI).
  12. Fulda, Nancy, Nathan Tibbetts, Zachary Brown, and David Wingate. 2017b. Harvesting common-sense navigational knowledge for robotics from uncurated text corpora. In Proceedings of the First Conference on Robot Learning.
  13. Gupta, Piyush, Inika Roy, Gunnika Batra, and Arun Ku- mar Dubey. 2021. Decoding emotions in text using glove embeddings. In 2021 International Conference on Computing, Communication, and Intelligent Sys- tems (ICCCIS), pages 36-40.
  14. Hamilton, William L., Jure Leskovec, and Dan Jurafsky. 2016. Diachronic word embeddings reveal statistical laws of semantic change. CoRR, abs/1605.09096.
  15. Jain, Vaibhav. 2020. GloVeInit at SemEval-2020 task 1: Using GloVe vector initialization for unsuper- vised lexical semantic change detection. In Proceed- ings of the Fourteenth Workshop on Semantic Evalua- tion, pages 208-213, Barcelona (online). International Committee for Computational Linguistics.
  16. Jansen, Stefan. 2017. Word and phrase translation with word2vec. CoRR, abs/1705.03127.
  17. Khalid, Usama, Aizaz Hussain, Muhammad Umair Ar- shad, Waseem Shahzad, and Mirza Omer Beg. 2021. Co-occurrences using fasttext embeddings for word similarity tasks in urdu. CoRR, abs/2102.10957.
  18. Khatri, Akshay and Pranav P. 2020. Sarcasm detection in tweets with BERT and GloVe embeddings. In Pro- ceedings of the Second Workshop on Figurative Lan- guage Processing, pages 56-60, Online. Association for Computational Linguistics.
  19. Khatua, Aparup, Apalak Khatua, and Erik Cambria. 2019. A tale of two epidemics: Contextual word2vec for classifying twitter streams during outbreaks. In- formation Processing & Management, 56(1):247-257.
  20. Kingma, Diederik P and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR (Poster).
  21. Kocmi, Tom and Ondřej Bojar. 2017. An exploration of word embedding initialization in deep-learning tasks. In Proceedings of the 14th International Conference on Natural Language Processing (ICON-2017), pages 56- 64, Kolkata, India. NLP Association of India.
  22. Lahiri, Shibamouli. 2014. Complexity of Word Colloca- tion Networks: A Preliminary Structural Analysis. In Proceedings of the Student Research Workshop at the 14th Conference of the European Chapter of the Asso- ciation for Computational Linguistics, pages 96-105, Gothenburg, Sweden. Association for Computational Linguistics.
  23. Lample, Guillaume, Alexis Conneau, Marc'Aurelio Ran- zato, Ludovic Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. In International Conference on Learning Representations.
  24. Lewis, Mike, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, Northern European Journal of Language Technology pages 7871-7880, Online. Association for Computa- tional Linguistics.
  25. Lin, Zhou, Qifeng Zhou, and Langcai Cao. 2021. Two- stage encoder for pointer-generator network with pretrained embeddings. In 2021 16th International Conference on Computer Science & Education (ICCSE), pages 524-529. IEEE.
  26. Liu, Nelson F., Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019a. Lin- guistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1073-1094, Minneapolis, Minnesota. Association for Computational Linguistics.
  27. Liu, Yinhan, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
  28. Mikolov, Tomas, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word represen- tations in vector space. arXiv preprint arXiv:1301.3781.
  29. Mikolov, Tomas, Ilya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013b. Distributed representa- tions of words and phrases and their composition- ality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, NIPS, pages 3111-3119. Curran Associates, Inc.
  30. Mikolov, Tomas, Wen-tau Yih, and Geoffrey Zweig. 2013c. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Con- ference of the North American Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies, pages 746-751, Atlanta, Georgia. Asso- ciation for Computational Linguistics.
  31. Minaee, Shervin, Nal Kalchbrenner, Erik Cambria, Nar- jes Nikzad, Meysam Chenaghlu, and Jianfeng Gao. 2021. Deep learning-based text classification: A com- prehensive review. ACM Comput. Surv., 54(3).
  32. Mitra, Bhaskar, Eric Nalisnick, Nick Craswell, and Rich Caruana. 2016. A dual embedding space model for document ranking. This paper is an extended evalu- ation and analysis of the model proposed in a poster to appear in WWW'16, April 11 -15, 2016, Montreal, Canada.
  33. Nalisnick, Eric, Mitra Bhaskar, Nick Craswell, and Rich Caruana. 2016. Improving document ranking with dual word embeddings. In WWW '16 Companion: Proceedings of the 25th International Conference Com- panion on World Wide Web, pages 83-84.
  34. Ni, Chien-Chun. 2015. Multiple choice question (MCQ) dataset. https://www3.cs.stonybrook.edu/ chni/post/mcq- dataset/.
  35. Pennington, Jeffrey, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word rep- resentation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Process- ing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1532-1543.
  36. Peters, Matthew, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computa- tional Linguistics.
  37. Peterson, Joshua C. 2019. OpenWebText. https://github.com/jcpeterson/openwebtext.
  38. Pickard, Thomas. 2020. Comparing word2vec and GloVe for automatic measurement of MWE composi- tionality. In Proceedings of the Joint Workshop on Mul- tiword Expressions and Electronic Lexicons, pages 95- 100, online. Association for Computational Linguis- tics.
  39. Pourdamghani, Nima, Marjan Ghazvininejad, and Kevin Knight. 2018. Using word vectors to improve word alignments for low resource machine transla- tion. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 524-528, New Or- leans, Louisiana. Association for Computational Lin- guistics.
  40. Premjith, B. 2019. Part of speech tagging ma- chine learning deep learning word2vec fastText. https://github.com/premjithb/Part-of-Speech- Tagging-Machine-Learning-Deep-Learning- Word2vec-fasttext.
  41. Qi, Ye, Devendra Sachan, Matthieu Felix, Sarguna Pad- manabhan, and Graham Neubig. 2018. When and why are pre-trained word embeddings useful for neu- ral machine translation? In Proceedings of the 2018 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Lan- guage Technologies, Volume 2 (Short Papers), pages
  42. Northern European Journal of Language Technology 529-535, New Orleans, Louisiana. Association for Computational Linguistics.
  43. Radford, Alec, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9.
  44. Robinson, Nathaniel, Zachary Brown, Timothy Sitze, and Nancy Fulda. 2021. Text classifications learned from language model hidden layers. In 2021 IEEE 19th World Symposium on Applied Machine Intelli- gence and Informatics (SAMI), pages 000207-000210. IEEE.
  45. Sabet, Masoud Jalili, Philipp Dufter, and Hinrich Schütze. 2020. Simalign: High quality word align- ments without parallel training data using static and contextualized embeddings. CoRR, abs/2004.08728.
  46. Speer, Robyn, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of gen- eral knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI'17, page 4444-4451. AAAI Press.
  47. Strubell, Emma, Ananya Ganesh, and Andrew McCal- lum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguis- tics, pages 3645-3650, Florence, Italy. Association for Computational Linguistics.
  48. Turney, Peter D. 2006. Similarity of semantic relations. Computational Linguistics, 32(3):379-416.
  49. Wilson, Theresa, Zornitsa Kozareva, Preslav Nakov, Alan Ritter, Sara Rosenthal, and Veselin Stoy- anov. 2013. Sentiment analysis in twitter. http://www.cs.york.ac.uk/semeval-2013/task2/.
  50. Yuan, Weizhe, Graham Neubig, and Pengfei Liu. 2021. BARTScore: Evaluating generated text as text gener- ation. Advances in Neural Information Processing Sys- tems, 34:27263-27277.
  51. Zhang, Tianyi, Varsha Kishore, Felix Wu, Kilian Q Wein- berger, and Yoav Artzi. 2019. BERTScore: Evaluating text generation with BERT. In International Confer- ence on Learning Representations.
  52. Zhu, Yudong, Di Zhou, Jinghui Xiao, Xin Jiang, Xiao Chen, and Qun Liu. 2020. HyperText: Endowing Fast- Text with hyperbolic geometry. In Findings of the As- sociation for Computational Linguistics: EMNLP 2020, pages 1166-1171, Online. Association for Computa- tional Linguistics.
  53. Zhu, Yukun, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. CoRR, abs/1506.06724.

Related papers

On Learning Word Embeddings From Linguistically Augmented Text Corpora
Chathurika Amarathunga

Proceedings of the 13th International Conference on Computational Semantics - Short Papers

Word embedding learning is a technique in Natural Language Processing (NLP) to map words into vector space representations, is one of the most popular research directions in modern NLP by virtue of its potential to boost the performance of many NLP downstream tasks. Nevertheless, most of the underlying word embedding methods such as word2vec and GloVe fail to produce high-quality representations if the text corpus is small and sparse. This paper proposes a method to generate effective word embeddings from limited data. Empirically, we show that the proposed model outperforms existing works for the classical word similarity task and for a domain-specific application.

View PDFchevron_right
Word2Vec: Optimal hyperparameters and their impact on natural language processing downstream tasks
Foteini Liwicki

Open Computer Science, 2022

Word2Vec is a prominent model for natural language processing tasks. Similar inspiration is found in distributed embeddings (word-vectors) in recent state-of-the-art deep neural networks. However, wrong combination of hyperparameters can produce embeddings with poor quality. The objective of this work is to empirically show that Word2Vec optimal combination of hyper-parameters exists and evaluate various combinations. We compare them with the publicly released, original Word2Vec embedding. Both intrinsic and extrinsic (downstream) evaluations are carried out, including named entity recognition and sentiment analysis. Our main contributions include showing that the best model is usually task-specific, high analogy scores do not necessarily correlate positively with F1 scores, and performance is not dependent on data size alone. If ethical considerations to save time, energy, and the environment are made, then relatively smaller corpora may do just as well or even better in some cases...

View PDFchevron_right
Towards Useful Word Embeddings
Petr Sojka

RASLAN, 2020

Since the seminal work of Mikolov et al. (2013), word vectors of log-bilinear models have found their way into many nlp applications and were extended with the positional model. Although the positional model improves accuracy on the intrinsic English word analogy task, prior work has neglected its evaluation on extrinsic end tasks, which correspond to real-world nlp applications. In this paper, we describe our first steps in evaluating positional weighting on the information retrieval, text classification, and language modeling extrinsic end tasks.

View PDFchevron_right
Word2Vec: Optimal Hyper-Parameters and Their Impact on NLP Downstream Tasks
Foteini Liwicki

arXiv (Cornell University), 2020

Word2Vec is a prominent model for natural language processing (NLP) tasks. Similar inspiration is found in distributed embeddings for new state-of-the-art (SotA) deep neural networks. However, wrong combination of hyper-parameters can produce poor quality vectors. The objective of this work is to empirically show optimal combination of hyper-parameters exists and evaluate various combinations. We compare them with the released, pretrained original word2vec model. Both intrinsic and extrinsic (downstream) evaluations, including named entity recognition (NER) and sentiment analysis (SA) were carried out. The downstream tasks reveal that the best model is usually task-specific, high analogy scores don't necessarily correlate positively with F1 scores and the same applies to the focus on data alone. Increasing vector dimension size after a point leads to poor quality or performance. If ethical considerations to save time, energy and the environment are made, then reasonably smaller corpora may do just as well or even better in some cases. Besides, using a small corpus, we obtain better WordSim scores, corresponding Spearman correlation and better downstream performances (with significance tests) compared to the original model, trained on a 100 billion-word corpus.

View PDFchevron_right
Training vs. Post-training Cross-lingual Word Embedding Approaches: A Comparative Study
Masood Ghayoomi

International Journal of Information Science and Management, 2023

This paper provides a comparative analysis of cross-lingual word embedding by studying the impact of different variables on the quality of the embedding models within the distributional semantics framework. Distributional semantics is a method for the semantic representation of words, phrases, sentences, and documents. This method aims at capturing as much information as possible from the contextual information in a vector space. The early study in this domain focused on monolingual word embedding. Further progress used cross-lingual data to capture the contextual semantic information across different languages. The main contribution of this research is to make a comparative study to find out the superior impact of the learning methods, supervised and unsupervised in training and post-training approaches in different embedding algorithms, to capture semantic properties of the words in cross-lingual embedding models to be applicable in tasks that deal with multi-languages, such as question retrieval. To this end, we study the cross-lingual embedding models created by BilBOWA, VecMap, and MUSE embedding algorithms along with the variables that impact the embedding models' quality, namely the size of the training data and the window size of the local context. In our study, we use the unsupervised monolingual Word2Vec embedding model as the baseline and evaluate the quality of embeddings on three data sets: Google analogy, mono-and cross-lingual words similar lists. We further investigated the impact of the embedding models in the question retrieval task.

View PDFchevron_right
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved