A Cascaded Classification Approach to Semantic Head Recognition

Advances in Natural …, 2004

Word fragments or n-grams have been widely used to perform different Natural Language Processing tasks such as information retrieval [2], document categorization [3], automatic summarization [4] or, even, genetic classification of languages . All these techniques share some common aspects such as: (1) documents are mapped to a vector space where n-grams are used as coordinates and their relative frequencies as vector weights, (2) many of them compute a context which plays a role similar to stop-word lists, and cosine distance is commonly used for document-to-document and query-to-document comparisons. blindLight is a new approach related to these classical n-gram techniques although it introduces two major differences: (1) Relative frequencies are no more used as vector weights but replaced by n-gram significances, and (2) cosine distance is abandoned in favor of a new metric inspired by sequence alignment techniques although not so computationally expensive. This new approach can be simultaneously used to perform document categorization and clustering, information retrieval, and text summarization. In this paper we will describe the foundations of such a technique and its application to both a particular categorization problem (i.e., language identification) and information retrieval tasks.

2020

Tokenization is a fundamental preprocessing step for almost all NLP tasks. In this paper, we propose efficient algorithms for the WordPiece tokenization used in BERT, from singleword tokenization to general text (e.g., sentence) tokenization. When tokenizing a single word, WordPiece uses a longest-matchfirst strategy, known as maximum matching. The best known algorithms so far are O(n2) (where n is the input length) or O(nm) (where m is the maximum vocabulary token length). We propose a novel algorithm whose tokenization complexity is strictly O(n). Our method is inspired by the Aho-Corasick algorithm. We introduce additional linkages on top of the trie built from the vocabulary, allowing smart transitions when the trie matching cannot continue. For general text, we further propose an algorithm that combines pre-tokenization (splitting the text into words) and our linear-time WordPiece method into a single pass. Experimental results show that our method is 8.2x faster than HuggingFa...

2019

Sample of related synsets extracted from WordNet. 3.3 Lexical chains construction. 3.4 Fictional pre-trained synset embeddings model. 3.5 Vector average for lexical chains centroids. 3.6 Cosine similarity between lexical chains elements and their centroids. 3.7 Final chains for FLLC II and FXLC II algorithms. 4.1 Dataset token details. WD10-English Wikipedia Dump 2010 (April); WD18-English Wikipedia Dump 2018 (January). 4.2 Spearman correlation score (ρ) for the RG65 benchmark. Highest results reported in bold face. 4.3 Spearman correlation score (ρ) for the MEN benchmark. The results of Chen et al. [32] and word2vec are reported in Mancini et al. [99] (MSSG/NP-MSSG). Highest results reported in bold face. 4.4 Spearman correlation score (ρ) for the WordSim353 benchmark. Huang et al. [73] results are reported in Neelakantan et al. [122] (MSSG/NP-MSSG). Highest results reported in bold face. 4.5 Spearman correlation score (ρ) for the SimLex999 benchmark. Chen et al. [32] and word2wec results are reported in Mancini et al. [99] (MSSG/NP-MSSG). Highest results reported in bold face. 4.6 Spearman correlation score for the MC28 benchmark. Highest results reported in bold face. 4.7 Spearman correlation score (ρ) for the SCWS benchmark. Huang et al. [73] results are reported in Neelakantan et al. [122] (MSSG/NP-MSSG). Highest results reported in bold face. 4.8 Technical details about the datasets after pre-processing 4.9 Grid-search configuration parameters. 4.10 Word embeddings used and their main characteristics. * For USE, Cer et al. [29] report its training data as a collection of sources from Wikipedia, web news, web question-answer pages discussion forums and Stanford Natural Language Inference corpus. viii 4.11 Classification accuracy for BOW approach against the proposed techniques for each classifier and dataset. Values in bold represent the best result of that row. Underlined values represent the best value for that dataset. K-NN-K Nearest Neighbors; RF-Random Forest; LR-Logistic Regression; SVM-Support Vector Machine; NB-Naïve Bayes. 4.12 Classification accuracy for word embeddings models against proposed techniques for each classifier and dataset. Values in bold represent the best result of that row. Underlined values represent the best value for that dataset. K-NN-K Nearest Neighbors; RF-Random Forest; LR-Logistic Regression; SVM-Support Vector Machine; NB-Naïve Bayes. 4.13 Classification accuracy for document embeddings models against proposed techniques for each classifier and dataset. Values in bold represent the best result of that row. Underlined values represent the best value for that dataset. K-NN-K Nearest Neighbors; RF-Random Forest; LR-Logistic Regression; SVM-Support Vector Machine; NB-Naïve Bayes. 6.1 Toy Example I-Ignoring single occurrences of synsets. 6.2 Toy Example II-Considering single occurrences of synsets having a 0 relative distance from themselves. 6.3 Toy Example III-Considering single occurrences of synsets and first synset of a chain having a 0 relative distance of themselves. 6.4 Wikipedia Dataset Details. 6.5 Experiments using lexical chains algorithms and traditional approaches. 6.6 Reuters-21578 Dataset topic distribution. 6.7 Sample Reuters-21578 (R8) Dataset topic distribution. 6.8 Silhouette values using Complete-Linkage for R8. 6.9 Silhouette values using Single-Linkage for R8. 6.10 Silhouette values using Average-Linkage for R8. 6.11 Distribution of synsets obtained through BSD and FLC. 6.12 MediaWiki categories sample. 6.13 Suggested synsets sample. ix LIST OF FIGURES Figure 2.1 Skip-gram and CBOW architectures. The skip-gram predicts context words given the current word, and CBOW predicts the word considering its context [109] 3.1 System architecture for extracting semantic features using MSSA and lexical chains algorithms. 3.2 System architecture of MSSA, MSSA-D and MSSA-NR. 3.3 MSSA-D illustration of the shortest path from w 1 to w 5 through their respective word-senses. 3.4 System architecture for building lexical chains. 4.1 Workflow of document classification. 4.2 Document classification ranking considering BOW and lexical chains techniques. 4.3 Document classification ranking considering state-of-the-art for word embeddings and lexical chains techniques. 4.4 Document classification ranking considering state-of-the-art for document embeddings and lexical chains techniques. 4.5 Accuracy for fixed lexical chains for variable chunk size and number of multiple recurrent passes for word embeddings models. 4.6 Accuracy for flexible lexical chains for multiple recurrent passes for word embeddings models. 4.7 Accuracy for fixed lexical chains for variable chunk size and number of multiple recurrent passes for document embbedings models. 4.8 Accuracy for flexible lexical chains for multiple recurrent passes for document embeddings models. 6.1 BSID through FSID and LSID evaluation process. 6.2 Flexible Lexical Chains example construction diagram. 6.3 Flexible to Fixed Lexical Chains construction diagram. 6.4 Fixed Lexical Chains construction diagram. 6.5 Scatter plot between mean individual silhouette and Adjusted Rand Index for proposed techniques (Table 6.5 data). 6.6 Consistency and Dissimilarity using Complete-Linkage for R8. 6.7 Consistency and Dissimilarity using Single-Linkage for R8. x 6.8 Consistency and Dissimilarity using Average-Linkage for R8. 6.9 Keyword survey average rankings. 6.10

… Symposium on Semantic …, 2005

In the biomedical domain, many systems for text mining and information extraction rely on basic morphological and syntactic analysis such as part-of-speech tagging or noun phrase (NP) chunking. Due to the lack of sufficient in-domain resources these systems often make use of NLP tools trained and evaluated on newspaper-language training sets. Scientific texts in the life sciences, however, differ from general language in the structure and complexity of noun phrases. Therefore, we tested the effects this domain change has on the performance of these systems. For this purpose, we compared three prototype chunking systems developed in research labs (all based on statistical learning methods) and one chunking system which is part of a commercial information extraction toolkit (based on manually supplied grammar specifications). Trained on PENN TREEBANK tagging and chunking annotations for newspapers, we ran these systems on the GENIA treebank which contains such annotations for biological abstracts taken from MED-LINE. We, first, observed a significant over-all loss in performance (on the order of 4%) and, second, found (with the exception of the SVM-based system) no significant difference between the performance of lab prototypes and the commerical chunker on GENIA data. Fortunately, the performance loss can also be partly remedied by few biomedical domain-specific adaptations.

Journal for Language Technology and Computational Linguistics, 2013

We present a novel method (“waste”) for the segmentation of text into tokens and sentences. Our approach makes use of a Hidden Markov Model for the detection of segment boundaries. Model parameters can be estimated from pre-segmented text which is widely available in the form of treebanks or aligned multi-lingual corpora. We formally define the waste boundary detection model and evaluate the system’s performance on corpora from various languages as well as a small corpus of computer-mediated communication.

Multiword detection is very difficult task in Language Processing. There has been a great change in the field of Natural Language Processing. Manual encoding of linguistic information is being challenged by automated corpus based learning methodologies for NLP with linguistic Knowledge. Although, Corpus based approaches have been successful in many different areas of Natural Language Processing. Multiword Detection using human evaluation method and machine evaluation method is a matter of discussion in present NLP research areas. Languages are not use not only to gather information but for communication as well. The problem is to take the information provided by the outside the world and translate the information into precise internal representation .This internal representation is called semantic representation. In this paper, we deals with how MWEs are generalized to optimized expensive evaluation of Multiword detection from a WordNet.

Lecture Notes in Computer Science, 2015

Since the professional technical literature include amounts of complex noun phrases, identifying those phrases has an important practical value for such tasks as machine translation. Through analysis of those phrases in Chinese-English bilingual sentence pairs from the aircraft technical publications, we present an annotation specification based on the existing specification to label those phrases and a method for the complex noun phrase identification. In addition to the basic features including the word and the part-of-speech, we incorporate the word clustering features trained by Brown clustering model and Word Vector Class (WVC) model on a large unlabeled data into the machine learning model. Experimental results indicate that the combination of different word clustering features and basic features can leverage system performance, and improve the F-score by 1.83% in contrast with the method only adding the basic features.

Cornell University - arXiv, 2022

Multiword expression (MWE) is a sequence of words which collectively present a meaning which is not derived from its individual words. MWEs detection is an important topic in different natural language processing (NLP) applications, including machine translation. Therefore, detecting MWEs in different domains is an important research topic. With this study, we explore state-of-the-art neural transformers in the task of detecting MWEs in flower and plant names. We evaluate different transformer models on a dataset created from Encyclopedia of Plants and Flower. We empirically show that neural transformers could outperform previous neural models like long-short term memory (LSTM) even in specific domains such as flower and plant names.

2004

This paper gives a brief overview of the results of our work during the Summer 2003 Workshop of the Center for Language and Speech Processing at the Johns Hopkins University in Baltimore Maryland. The goal of the project was to determine the feasibility of extending named entity recognition to common nouns and determine whether or not it is possible to assign automatically a predetermined set of semantic tags and approach human performance in the task.

Using Neural Networks (NN), Natural Language Processing (NLP) or a dictionary individually to detect noun and verb multiword expressions (MWEs) in an English sentence often results in low accuracy and precision. Our approach combines pre-trained, self-supervised, transformer-based neural networks based on BERT and ALBERT with part-of-speech and sentence dependency, Natural Language Processing algorithms as well as the Unified Compliance, patented dictionary. Experiments were performed on the UCF GRC dataset and publicly available Detecting Minimal Semantic Units and their Meanings (DiM-SUM) dataset . As a result, our approach achieves an F1 score of 73.52% for MWE identification which beats the previous state of the art, models F1 score of 40.76% .