Academia.eduAcademia.edu

Figure 3 – uploaded by Luke Dickens

See full PDFdownloadDownload figure
Figure 3: Spearman’s p between embedding distances and word similarity scores on the SimLex999 dataset on our three vocabulary sizes with a constant data aug- mentation ratio of 25%. w2v shows the performance of the unaltered Word2Vec embeddings.

Figure 3 Spearman’s p between embedding distances and word similarity scores on the SimLex999 dataset on our three vocabulary sizes with a constant data aug- mentation ratio of 25%. w2v shows the performance of the unaltered Word2Vec embeddings.

Related Figures (3)

Figure 1: Synonym augmentation pipeline context size C' = 5, k = 5 negative samples (which the paper suggests for smaller datasets), trained for 10 epochs with a learning rate of 0.01 since that re- sulted in a convergent training phase*, and a batch size of 10 to speed up training. We skipped the subsampling phase described in (Mikolov et al., 2013a) because our dataset was not large enough to warrant an additional speed up, and because our data augmentation strategy already down-weights stop words by leaving them out of the augmentation process. We trained our word embeddings with a fixed augmentation ratio of r = 0.25 (i.e. the train- ing data consisted of 75% natural data and 25% synonym-augmented data), which we empirically found to be the best performing ratio.
Figure 1: Synonym augmentation pipeline context size C' = 5, k = 5 negative samples (which the paper suggests for smaller datasets), trained for 10 epochs with a learning rate of 0.01 since that re- sulted in a convergent training phase*, and a batch size of 10 to speed up training. We skipped the subsampling phase described in (Mikolov et al., 2013a) because our dataset was not large enough to warrant an additional speed up, and because our data augmentation strategy already down-weights stop words by leaving them out of the augmentation process. We trained our word embeddings with a fixed augmentation ratio of r = 0.25 (i.e. the train- ing data consisted of 75% natural data and 25% synonym-augmented data), which we empirically found to be the best performing ratio.
Figure 2: Spearman’s p for varying ratios of data aug- mentation (0%, 2%, 3.5%, 6%, 10%, 16%, 25%, 37%, 50%, 64%) and for the unaltered Word2Vec embed- dings (w2v), between embedding distances and word similarity or relatedness scores for three datasets: Sim- Lex999, WordSim353-similarity, and WordSim353- relatedness. Table 1: KNN+WMD accuracies for the 20 News Groups dataset (intervals for a confidence level of 95% are given in parenthesis). W2V (base) is the unaltered Word2Vec model pruned to different vocabulary sizes. W2V init. and Rand. init. refer to the different initialisation points, and are divided into non-augmented (NoSyn) and augmented (Syn) training data. Voc-W2V is the result from the pre-trained Word2Vec embeddings with the full 3,000,000 word vocabulary, which in the original WMD paper by Kusner et al. (2015) is reported to have an accuracy of .73.
Figure 2: Spearman’s p for varying ratios of data aug- mentation (0%, 2%, 3.5%, 6%, 10%, 16%, 25%, 37%, 50%, 64%) and for the unaltered Word2Vec embed- dings (w2v), between embedding distances and word similarity or relatedness scores for three datasets: Sim- Lex999, WordSim353-similarity, and WordSim353- relatedness. Table 1: KNN+WMD accuracies for the 20 News Groups dataset (intervals for a confidence level of 95% are given in parenthesis). W2V (base) is the unaltered Word2Vec model pruned to different vocabulary sizes. W2V init. and Rand. init. refer to the different initialisation points, and are divided into non-augmented (NoSyn) and augmented (Syn) training data. Voc-W2V is the result from the pre-trained Word2Vec embeddings with the full 3,000,000 word vocabulary, which in the original WMD paper by Kusner et al. (2015) is reported to have an accuracy of .73.
Figure 4: Word embedding pairwise cosine distance mean and standard deviation for three sets of word pairs: syn- onyms (syn), contextual (nat), and random (rand). As in our other plots, rand_ and w2vi_ refer to the initialisation, s_ and ns_ correspond to augmented and non-augmented respectively, v_ indicates the vocabulary size, and r_ the augmentation ratio.
Figure 4: Word embedding pairwise cosine distance mean and standard deviation for three sets of word pairs: syn- onyms (syn), contextual (nat), and random (rand). As in our other plots, rand_ and w2vi_ refer to the initialisation, s_ and ns_ correspond to augmented and non-augmented respectively, v_ indicates the vocabulary size, and r_ the augmentation ratio.
Related topics:
Computer Science
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved