Academia.eduAcademia.edu

Outline

Title
Abstract
Figures
Introduction
Related Work
Background: Graph Convolutional Networks
Wordgcn Overview
Dataset and Training
Evaluation Method
Results
Ablation Results
Qualitative Results
Conclusion
References
All Topics
Computer Science

Graph Convolutional Networks based Word Embeddings

Profile image of Prateek YadavPrateek Yadav

2018, ArXiv

Abstract

Recently, word embeddings have been widely adopted across several NLP applications. However, most word embedding methods solely rely on linear context and do not provide a framework for incorporating word relationships like hypernym, nmod in a principled manner. In this paper, we propose WordGCN, a Graph Convolution based word representation learning approach which provides a framework for exploiting multiple types of word relationships. WordGCN operates at sentence as well as corpus level and allows to incorporate dependency parse based context in an efficient manner without increasing the vocabulary size. To the best of our knowledge, this is the first approach which effectively incorporates word relationships via Graph Convolutional Networks for learning word representations. Through extensive experiments on various intrinsic and extrinsic tasks, we demonstrate WordGCN's effectiveness over existing word embedding approaches. We make WordGCN's source code available to enco...

Figures (6)
Figure 1: Overview of WordGCN. WordGCN utilizes sentence level word co-occurrence relationships and directed semantic information for learning word embeddings through S-GCN component. These are further improved by RF-GCN, which incor- porates corpus level synonym relationships. Each word has two embeddings — neighborhood and target, for both components. W and W’ consist of target embeddings of each word in the vocabulary which is used to compute the final softmax scores. Please refer Section 4 for details.
Figure 1: Overview of WordGCN. WordGCN utilizes sentence level word co-occurrence relationships and directed semantic information for learning word embeddings through S-GCN component. These are further improved by RF-GCN, which incor- porates corpus level synonym relationships. Each word has two embeddings — neighborhood and target, for both components. W and W’ consist of target embeddings of each word in the vocabulary which is used to compute the final softmax scores. Please refer Section 4 for details.
Table 1: Evaluation on three intrinsic tasks: word similarity (spearman correlation), concept categorization (cluster purity), and word analogy (spearman correlation). WordGCN(-Sem) denotes WordGCN without using any information from semantic knowledge sources. Overall, WordGCN either outperforms or performs competitively compared to other existing approaches. Please refer to Section 7.1 for more details.
Table 1: Evaluation on three intrinsic tasks: word similarity (spearman correlation), concept categorization (cluster purity), and word analogy (spearman correlation). WordGCN(-Sem) denotes WordGCN without using any information from semantic knowledge sources. Overall, WordGCN either outperforms or performs competitively compared to other existing approaches. Please refer to Section 7.1 for more details.
Table 2: Evaluation on extrinsic tasks: question classi- fication, news categorization, named entity recognition, parts-of-speech tagging, and neural co-reference resolution. WordGCN outperforms all existing approaches on four out of five tasks. Refer Section 7.2 for details. Figure 2: Average scores on similarity and categorization tasks. Similar trends are observed on analogy task. Dep stands for only dependency based context in WordGCN. Similarly Con and W stand for linear context and WordNet information respectively. Refer Section 7.3 for details.
Table 2: Evaluation on extrinsic tasks: question classi- fication, news categorization, named entity recognition, parts-of-speech tagging, and neural co-reference resolution. WordGCN outperforms all existing approaches on four out of five tasks. Refer Section 7.2 for details. Figure 2: Average scores on similarity and categorization tasks. Similar trends are observed on analogy task. Dep stands for only dependency based context in WordGCN. Similarly Con and W stand for linear context and WordNet information respectively. Refer Section 7.3 for details.
Table 3: Comparison of average similarity scores of RF- GCN against other methods on incorporating semantic knowledge in pre-trained embeddings. Refer to Section 7.4 for details.
Table 3: Comparison of average similarity scores of RF- GCN against other methods on incorporating semantic knowledge in pre-trained embeddings. Refer to Section 7.4 for details.
Table 4: Comparison of words close to the target word ac- cording to different word embedding methods ment whereas WordGCN is also able to capture the other sense of bank i.e. riverbank. For the word grave, we find WordGCN captures its sense as a part of graveyard (topi- cal similarity) and also captures its other meaning i.e. to be serious (functional similarity). This shows that WordGCN captures multiple, diverse connotations of a word as well as functional and topical similarities.
Table 4: Comparison of words close to the target word ac- cording to different word embedding methods ment whereas WordGCN is also able to capture the other sense of bank i.e. riverbank. For the word grave, we find WordGCN captures its sense as a part of graveyard (topi- cal similarity) and also captures its other meaning i.e. to be serious (functional similarity). This shows that WordGCN captures multiple, diverse connotations of a word as well as functional and topical similarities.

Related papers

Graph-based Dynamic Word Embeddings
Ziran Liang

Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence

As time goes by, language evolves with word semantics changing. Unfortunately, traditional word embedding methods neglect the evolution of language and assume that word representations are static. Although contextualized word embedding models can capture the diverse representations of polysemous words, they ignore temporal information as well. To tackle the aforementioned challenges, we propose a graph-based dynamic word embedding (GDWE) model, which focuses on capturing the semantic drift of words continually. We introduce word-level knowledge graphs (WKGs) to store short-term and long-term knowledge. WKGs can provide rich structural information as supplement of lexical information, which help enhance the word embedding quality and capture semantic drift quickly. Theoretical analysis and extensive experiments validate the effectiveness of our GDWE on dynamic word embedding learning.

View PDFchevron_right
Improving Word Representations via Global Context and Multiple Word Prototypes
Christopher D Manning

Unsupervised word representations are very useful in NLP tasks both as inputs to learning algorithms and as extra word features in NLP systems. However, most of these models are built with only local context and one representation per word. This is problematic because words are often polysemous and global context can also provide useful information for learning word meanings. We present a new neural network architecture which 1) learns word embeddings that better capture the semantics of words by incorporating both local and global document context, and 2) accounts for homonymy and polysemy by learning multiple embeddings per word. We introduce a new dataset with human judgments on pairs of words in sentential context, and evaluate our model on it, showing that our model outperforms competitive baselines and other neural language models. 1

View PDFchevron_right
Exploring the Combination of Contextual Word Embeddings and Knowledge Graph Embeddings
Kelvin Han

ArXiv, 2020

"Classical" word embeddings, such as Word2Vec, have been shown to capture the semantics of words based on their distributional properties. However, their ability to represent the different meanings that a word may have is limited. Such approaches also do not explicitly encode relations between entities, as denoted by words. Embeddings of knowledge bases (KB) capture the explicit relations between entities denoted by words, but are not able to directly capture the syntagmatic properties of these words. To our knowledge, recent research have focused on representation learning that augment the strengths of one with the other. In this work, we begin exploring another approach using contextual and KB embeddings jointly at the same level and propose two tasks-an entity typing and a relation typing task-that evaluate the performance of contextual and KB embeddings. We also evaluated a concatenated model of contextual and KB embeddings with these two tasks, and obtain conclusive results on the first task. We hope our work may contribute as a basis for models and datasets that develop in the direction of this approach.

View PDFchevron_right
Together is Better: Hybrid Recommendations Combining Graph Embeddings and Contextualized Word Representations
Giovanni Semeraro

Fifteenth ACM Conference on Recommender Systems

In this paper, we present a hybrid recommendation framework based on the combination of graph embeddings and contextual word representations. Our approach is based on the intuition that each of the above mentioned representation models heterogeneous (and equally important) information, that is worth to be taken into account to generate a recommendation. Accordingly, we propose a strategy to combine both the features, which is based on the following steps: first, we separately generate graph embeddings and contextual word representations by exploiting state-of-the-art techniques. Next, these embeddings are used to feed a deep architecture that learns a hybrid representation based on the combination of the single groups of features. Finally, we exploit the resulting embedding to identify suitable recommendations. In the experimental session, we evaluate the effectiveness of our strategy on two datasets and results show that the use of a hybrid representation leads to an improvement of the predictive accuracy. Moreover, our approach overcomes several competitive baselines, thus confirming the validity of this work. CCS CONCEPTS • Information systems → Recommender systems; • Computing methodologies → Natural language processing.

View PDFchevron_right
Enhanced word embeddings using multi-semantic representation through lexical chains
Terry Ruas

Information Sciences, 2020

The relationship between words in a sentence often tells us more about the underlying semantic content of a document than its actual words, individually. In this work, we propose two novel algorithms, called Flexible Lexical Chain II and Fixed Lexical Chain II. These algorithms combine the semantic relations derived from lexical chains, prior knowledge from lexical databases, and the robustness of the distributional hypothesis in word embeddings as building blocks forming a single system. In short, our approach has three main contributions: (i) a set of techniques that fully integrate word embeddings and lexical chains; (ii) a more robust semantic representation that considers the latent relation between words in a document; and (iii) lightweight word embeddings models that can be extended to any natural language task. We intend to assess the knowledge of pre-trained models to evaluate their robustness in document classification task. The proposed techniques are tested against seven word embeddings algorithms using five different machine learning classifiers over six scenarios in the document classification task. Our results show the integration between lexical chains and word embeddings representations sustain state-of-the-art results, even against more complex systems.

View PDFchevron_right
A Survey On Neural Word Embeddings
Selma Tekir

ArXiv, 2021

Understanding human language has been a sub-challenge on the way of intelligent machines. The study of meaning in natural language processing (NLP) relies on the distributional hypothesis where language elements get meaning from the words that co-occur within contexts. The revolutionary idea of distributed representation for a concept is close to the working of a human mind in that the meaning of a word is spread across several neurons, and a loss of activation will only slightly affect the memory retrieval process. Neural word embeddings transformed the whole field of NLP by introducing substantial improvements in all NLP tasks. In this survey, we provide a comprehensive literature review on neural word embeddings. We give theoretical foundations and describe existing work by an interplay between word embeddings and language modeling. We provide broad coverage on neural word embeddings, including early word embeddings, embeddings targeting specific semantic relations, sense embeddi...

View PDFchevron_right
Embedding Semantic Relations into Word Representations
Ken-ichi Kawarabayashi

Learning representations for semantic relations is important for various tasks such as analogy detection , relational search, and relation classification. Although there have been several proposals for learning representations for individual words, learning word representations that explicitly capture the semantic relations between words remains under developed. We propose an unsupervised method for learning vector representations for words such that the learnt representations are sensitive to the semantic relations that exist between two words. First, we extract lexical patterns from the co-occurrence contexts of two words in a corpus to represent the semantic relations that exist between those two words. Second, we represent a lexical pattern as the weighted sum of the representations of the words that co-occur with that lexical pattern. Third, we train a binary classifier to detect relationally similar vs. non-similar lexical pattern pairs. The proposed method is unsupervised in ...

View PDFchevron_right
Learning Graph Embeddings from WordNet-based Similarity Measures
Alexander Panchenko, Andrey Kutuzov

Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019), 2019

We present path2vec, a new approach for learning graph embeddings that relies on structural measures of pairwise node similarities. The model learns representations for nodes in a dense space that approximate a given user-defined graph distance measure, such as e.g. the shortest path distance or distance measures that take information beyond the graph structure into account. Evaluation of the proposed model on semantic similarity and word sense disambiguation tasks, using various WordNet-based similarity measures, show that our approach yields competitive results, outperforming strong graph embedding baselines. The model is computationally efficient, being orders of magnitude faster than the direct computation of graph-based distances.

View PDFchevron_right
Can Network Embedding of Distributional Thesaurus Be Combined with Word Vectors for Better Representation?
Abhik Jana

Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 2018

Distributed representations of words learned from text have proved to be successful in various natural language processing tasks in recent times. While some methods represent words as vectors computed from text using predictive model (Word2vec) or dense count based model (GloVe), others attempt to represent these in a distributional thesaurus network structure where the neighborhood of a word is a set of words having adequate context overlap. Being motivated by recent surge of research in network embedding techniques (DeepWalk, LINE, node2vec etc.), we turn a distributional thesaurus network into dense word vectors and investigate the usefulness of distributional thesaurus embedding in improving overall word representation. This is the first attempt where we show that combining the proposed word representation obtained by distributional thesaurus embedding with the state-of-the-art word representations helps in improving the performance by a significant margin when evaluated against NLP tasks like word similarity and relatedness, synonym detection, analogy detection. Additionally, we show that even without using any handcrafted lexical resources we can come up with representations having comparable performance in the word similarity and relatedness tasks compared to the representations where a lexical resource has been used.

View PDFchevron_right
Towards Lexical Chains for Knowledge-Graph-basedWord Embeddings
Petya Osenova

RANLP 2017 - Recent Advances in Natural Language Processing Meet Deep Learning, 2017

Word vectors with varying dimensionalities and produced by different algorithms have been extensively used in NLP. The corpora that the algorithms are trained on can contain either natural language text (e.g. Wikipedia or newswire articles) or artificially-generated pseudo corpora due to natural data sparseness. We exploit Lexical Chain based templates over Knowledge Graph for generating pseudo-corpora with controlled linguistic value. These corpora are then used for learning word embeddings. A number of experiments have been conducted over the following test sets: WordSim353 Similarity, WordSim353 Relatedness and SimLex-999. The results show that, on the one hand, the incorporation of many-relation lexical chains improves results, but on the other hand, unrestricted-length chains remain difficult to handle with respect to their huge quantity.

View PDFchevron_right

References (25)

  1. Almuhareb, A. 2006. Attributes in lexical ac- quisition.
  2. Alsuhaibani et al. 2018] Alsuhaibani, M.; Bollegala, D.; Maehara, T.; and Kawarabayashi, K.-i. 2018. Jointly learning word embed- dings using a corpus and a knowledge base. PLOS ONE 13(3):1- 26.
  3. Baker, Fillmore, and Lowe 1998] Baker, C. F.; Fillmore, C. J.; and Lowe, J. B. 1998. The berkeley framenet project. In ACL, ACL '98, 86-90. Stroudsburg, PA, USA: ACL.
  4. Baroni and Lenci 2010] Baroni, M., and Lenci, A. 2010. Distribu- tional memory: A general framework for corpus-based semantics. Comput. Linguist. 36(4):673-721.
  5. Baroni and Lenci 2011] Baroni, M., and Lenci, A. 2011. How we blessed distributional semantic evaluation. In GEMS, GEMS '11, 1-10. Stroudsburg, PA, USA: ACL.
  6. Bastings et al. 2017] Bastings, J.; Titov, I.; Aziz, W.; Marcheg- giani, D.; and Simaan, K. 2017. Graph convolutional encoders for syntax-aware neural machine translation. In EMNLP, 1957-1967. ACL. [Bengio et al. 2003] Bengio, Y.; Ducharme, R.; Vincent, P.; and Janvin, C. 2003. A neural probabilistic language model. JMLR 3:1137-1155.
  7. Bengio, Courville, and Vincent 2013] Bengio, Y.; Courville, A.; and Vincent, P. 2013. Representation learning: A review and new perspectives. IEEE TPAMI 35(8):1798-1828.
  8. Bronstein et al. 2017] Bronstein, M. M.; Bruna, J.; LeCun, Y.; Szlam, A.; and Vandergheynst, P. 2017. Geometric deep learning: Going beyond euclidean data. IEEE Signal Processing Magazine 34(4):18-42.
  9. Bruna et al. 2013] Bruna, J.; Zaremba, W.; Szlam, A.; and LeCun, Y. 2013. Spectral networks and locally connected networks on graphs. CoRR abs/1312.6203.
  10. Bruni, Tran, and Baroni 2014] Bruni, E.; Tran, N. K.; and Baroni, M. 2014. Multimodal distributional semantics. J. Artif. Int. Res. 49(1):1-47. [Collobert et al. 2011] Collobert, R.; Weston, J.; Bottou, L.; Karlen, M.; Kavukcuoglu, K.; and Kuksa, P. 2011. Natural language pro- cessing (almost) from scratch. JMLR 12:2493-2537.
  11. Defferrard, Bresson, and Vandergheynst 2016] Defferrard, M.; Bresson, X.; and Vandergheynst, P. 2016. Convolutional neural networks on graphs with fast localized spectral filtering. CoRR abs/1606.09375.
  12. Dernoncourt, Lee, and Szolovits 2017] Dernoncourt, F.; Lee, J. Y.; and Szolovits, P. 2017. NeuroNER: an easy-to-use program for named-entity recognition based on neural networks. EMNLP. [Faruqui et al. 2014] Faruqui, M.; Dodge, J.; Jauhar, S. K.; Dyer, C.; Hovy, E. H.; and Smith, N. A. 2014. Retrofitting word vectors to semantic lexicons. CoRR abs/1411.4166. [Finkelstein et al. 2001] Finkelstein, L.; Gabrilovich, E.; Matias, Y.; Rivlin, E.; Solan, Z.; Wolfman, G.; and Ruppin, E. 2001. Placing search in context: The concept revisited. In WWW 2001. [Glorot and Bengio 2010] Glorot, X., and Bengio, Y. 2010. Un- derstanding the difficulty of training deep feedforward neural net- works. In AISTATS. [Gutmann and Hyvärinen 2012] Gutmann, M. U., and Hyvärinen, A. 2012. Noise-contrastive estimation of unnormalized statisti- cal models, with applications to natural image statistics. J. Mach. Learn. Res. 13(1):307-361.
  13. Harris 1954] Harris, Z. S. 1954. Distributional structure. WORD. [Jastrzebski, Lesniak, and Czarnecki ] Jastrzebski, S.; Lesniak, D.; and Czarnecki, W. M. How to evaluate word embeddings? on importance of data efficiency and simple supervised tasks. abs/1702.02170.
  14. Jean et al. 2015] Jean, S.; Cho, K.; Memisevic, R.; and Bengio, Y. 2015. On using very large target vocabulary for neural machine translation. In IJCNLP, 1-10. ACL.
  15. Ji et al. 2015] Ji, S.; Yun, H.; Yanardag, P.; Matsushima, S.; and Vishwanathan, S. V. N. 2015. Wordrank: Learning word embed- dings via robust ranking. CoRR abs/1506.02761. [Kingma and Ba 2014] Kingma, D., and Ba, J. 2014. Adam: A method for stochastic optimization. [Kipf and Welling 2016] Kipf, T. N., and Welling, M. 2016. Semi- supervised classification with graph convolutional networks. CoRR abs/1609.02907. [Komninos and Manandhar 2016] Komninos, A., and Manandhar, S. 2016. Dependency based embeddings for sentence classifica- tion tasks.
  16. Lee, He, and Zettlemoyer ] Lee, K.; He, L.; and Zettlemoyer, L. Higher-order coreference resolution with coarse-to-fine inference. In NAACL 2018. [Levy and Goldberg 2014] Levy, O., and Goldberg, Y. 2014. Dependency-based word embeddings. In ACL, 302-308. ACL. [Li and Roth 2006] Li, X., and Roth, D. 2006. Learning question classifiers: The role of semantic information. NLE 12(3):229-249.
  17. Li et al. 2018] Li, C.; Li, J.; Song, Y.; and Lin, Z. 2018. Training and evaluating improved dependency-based word embeddings. In AAAI. [Ma and Hovy 2016] Ma, X., and Hovy, E. H. 2016. End-to- end sequence labeling via bi-directional lstm-cnns-crf. CoRR abs/1603.01354. [Maas et al. 2011] Maas, A. L.; Daly, R. E.; Pham, P. T.; Huang, D.; Ng, A. Y.; and Potts, C. 2011. Learning word vectors for sentiment analysis. In ACL, HLT '11, 142-150. Stroudsburg, PA, USA: ACL. [Manning et al. 2014] Manning, C. D.; Surdeanu, M.; Bauer, J.; Finkel, J.; Bethard, S. J.; and McClosky, D. 2014. The Stanford CoreNLP natural language processing toolkit. In ACL, 55-60. [Marcheggiani and Titov 2017] Marcheggiani, D., and Titov, I. 2017. Encoding sentences with graph convolutional networks for semantic role labeling. In EMNLP, 1506-1515. ACL. [Mikolov et al. 2013] Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G.; and Dean, J. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th In- ternational Conference on Neural Information Processing Systems -Volume 2, NIPS'13, 3111-3119. USA: Curran Associates Inc. [Miller 1995] Miller, G. A. 1995. Wordnet: A lexical database for english. Commun. ACM 38(11):39-41.
  18. Mnih and Hinton ] Mnih, A., and Hinton, G. A scalable hierarchi- cal distributed language model. In NIPS, NIPS'08. [Morin and Bengio 2005] Morin, F., and Bengio, Y. 2005. Hierar- chical probabilistic neural network language model. In AISTATS. [Mrkšić et al. 2016] Mrkšić, N.; Ó Séaghdha, D.; Thomson, B.; Gašić, M.; Rojas-Barahona, L. M.; Su, P.-H.; Vandyke, D.; Wen, T.-H.; and Young, S. 2016. Counter-fitting word vectors to linguis- tic constraints. In ACL, 142-148. ACL. [Nguyen and Grishman 2018] Nguyen, T., and Grishman, R. 2018. Graph convolutional networks with argument-aware pooling for event detection.
  19. Pavlick et al. 2015] Pavlick, E.; Rastogi, P.; Ganitkevitch, J.; Van Durme, B.; and Callison-Burch, C. 2015. Ppdb 2.0: Better paraphrase ranking, fine-grained entailment relations, word embed- dings, and style classification. In ACL, 425-430. ACL.
  20. Pennington, Socher, and Manning 2014] Pennington, J.; Socher, R.; and Manning, C. D. 2014. Glove: Global vectors for word representation. In EMNLP, 1532-1543.
  21. Pradhan et al. ] Pradhan, S.; Moschitti, A.; Xue, N.; Uryupina, O.; and Zhang, Y. Conll-2012 shared task: Modeling multilingual un- restricted coreference in ontonotes. In EMNLP-CONLL, CoNLL '12. [Radinsky et al. ] Radinsky, K.; Agichtein, E.; Gabrilovich, E.; and Markovitch, S. A word at a time: Computing word relatedness using temporal semantic analysis. In WWW 2011. [Reimers and Gurevych 2017] Reimers, N., and Gurevych, I. 2017. Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging. In EMNLP, 338- 348.
  22. Rumelhart, Hinton, and Williams 1988] Rumelhart, D. E.; Hinton, G. E.; and Williams, R. J. 1988. Neurocomputing: Foundations of research. Cambridge, MA, USA: MIT Press. chapter Learning Representations by Back-propagating Errors, 696-699.
  23. Scarselli et al. 2009] Scarselli, F.; Gori, M.; Tsoi, A. C.; Hagen- buchner, M.; and Monfardini, G. 2009. The graph neural network model. IEEE Transactions on Neural Networks 20(1):61-80.
  24. Socher et al. 2013] Socher, R.; Bauer, J.; Manning, C. D.; and Ng, A. Y. 2013. Parsing With Compositional Vector Grammars. In ACL. [Song, Lee, and Xia 2017] Song, Y.; Lee, C.-J.; and Xia, F. 2017. Learning word representations with regularization from prior knowledge. In CONLL 2017.
  25. Tjong Kim Sang and De Meulder ] Tjong Kim Sang, E. F., and De Meulder, F. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In HLT-NAACL, CONLL '03. [Vashishth et al. 2018] Vashishth, S.; Dasgupta, S. S.; Ray, S. N.; and Talukdar, P. 2018. Dating documents using graph convolution networks. In ACL, 1605-1615. ACL. [Yu and Dredze 2014] Yu, M., and Dredze, M. 2014. Improving lexical embeddings with semantic knowledge. In ACL, 545-550. ACL.

Related papers

Incorporating Syntactic and Semantic Information in Word Embeddings using Graph Convolutional Networks
prateek yadav

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

Word embeddings have been widely adopted across several NLP applications. Most existing word embedding methods utilize sequential context of a word to learn its embedding. While there have been some attempts at utilizing syntactic context of a word, such methods result in an explosion of the vocabulary size. In this paper, we overcome this problem by proposing SynGCN, a flexible Graph Convolution based method for learning word embeddings. SynGCN utilizes the dependency context of a word without increasing the vocabulary size. Word embeddings learned by SynGCN outperform existing methods on various intrinsic and extrinsic tasks and provide an advantage when used with ELMo. We also propose SemGCN, an effective framework for incorporating diverse semantic knowledge for further enhancing learned word representations. We make the source code of both models available to encourage reproducible research.

View PDFchevron_right
Words are Vectors, Dependencies are Matrices: Learning Word Embeddings from Dependency Graphs
Paula Czarnowska

Proceedings of the 13th International Conference on Computational Semantics - Long Papers, 2019

Distributional Semantic Models (DSMs) construct vector representations of word meanings based on their contexts. Typically, the contexts of a word are defined as its closest neighbours, but they can also be retrieved from its syntactic dependency relations. In this work, we propose a new dependencybased DSM. The novelty of our model lies in associating an independent meaning representation, a matrix, with each dependency-label. This allows it to capture specifics of the relations between words and contexts, leading to good performance on both intrinsic and extrinsic evaluation tasks. In addition to that, our model has an inherent ability to represent dependency chains as products of matrices which provides a straightforward way of handling further contexts of a word.

View PDFchevron_right
Learning Word Representations from Relational Graphs
Ken-ichi Kawarabayashi

Attributes of words and relations between two words are central to numerous tasks in Artificial Intelligence such as knowledge representation, similarity measurement, and analogy detection. Often when two words share one or more attributes in common, they are connected by some semantic relations. On the other hand, if there are numerous semantic relations between two words, we can expect some of the attributes of one of the words to be inherited by the other. Motivated by this close connection between attributes and relations, given a relational graph in which words are interconnected via numerous semantic relations, we propose a method to learn a latent representation for the individual words. The proposed method considers not only the co-occurrences of words as done by existing approaches for word representation learning, but also the semantic relations in which two words co-occur. To evaluate the accuracy of the word representations learnt using the proposed method, we use the learnt word representations to solve semantic word analogy problems. Our experimental results show that it is possible to learn better word representations by using semantic semantics between words.

View PDFchevron_right
Using Graph Convolutional Neural Networks for NLP tasks
Sanchit Sinha

2020

In recent times, Graph Neural Networks are proposed as a paradigm shifting methodology in solving standard Natural Language Processing and Computer Vision tasks. Graph Neural Networks provide a much more generalised method of representing word embeddings, image pixels, etc. Analogous to the standard Deep Learning literature, there have been multiple variants of GNNs proposed such as Convolutional GNNs, Recurrent GNNs, AutoEncoder GNNs, etc. Similar to using Convolution Networks in text classification, Graph Convolution Networks can also be used in text classification tasks. In this project, we plan to use Graph Convolutional Neural Networks to solve the open problem of Text Classification and Sentiment Analysis. Using a baseline model from Yao et al., we make certain modifications in the model training procedure keeping the architecture similar to the one proposed. Results on benchmark datasets show that the results improve from what are reported. ACM Reference Format: Sanchit Sinha...

View PDFchevron_right
Word, Sense, and Graph Embeddings
Ronald Denaux

A Practical Guide to Hybrid Natural Language Processing, 2020

Distributed word representations in the form of dense vectors, known as word embeddings, are the basic building blocks for machine-learning based natural language processing. Such embeddings play an important role in tasks such as part-of-speech tagging, chunking, named entity recognition, and semantic role labeling, as well as downstream tasks including sentiment analysis and more in general text classification. However, early word embeddings were static contextindependent representations that fail to capture multiple meanings for polysemous words. This chapter presents an overview of such traditional word embeddings, but also of alternative approaches that have been proposed to produce sense and concept embeddings using disambiguated corpora or directly from knowledge graphs. As a result, this chapter serves as a conceptual framework for the rest of book.

View PDFchevron_right

Related topics

  • Computer Science
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved