Assamese Word Sense Disambiguation using Genetic Algorithm

Word Sense Disambiguation (WSD) is the process of identifying the proper sense of an ambiguous word depending on the particular context. It is to find the accurate sense si among the set of senses {s1, s2, …, sn}. This task was motivated by its interpretation in various Natural Language Processing (NLP) applications like IR, MT, QA, TC, SP etc. In this paper, machine learning technique - Naïve Bayes Classifier was used for automatic disambiguation task. Training data was prepared with sense annotated features. For preparing sense annotated data we took help of the sense inventory. Currently, about 160 ambiguous words are present in the sense inventory derived from 18K and 25K words from Assamese Corpus and WordNet. The system is implemented in two phases. In the first phase, a total of 2.7K sense annotated training data and 800 test data were taken and a result of 71% accuracy was found. Analyzing the result depicts that accuracy improves as the training data size gradually increases and by the learned model generated in the previous iteration. In second phase we manually validate the outcomes of first-phase and we add those clean sense tagged data to previous training data set. Than we train our system with our incresing training data (3.5K) which enhance the result accuracy. An iterative learning is adopted by the system and more accuracy of 7% is achieved. This paper aims to implement Assamese WSD system by NB classifier using lexical features and enhancement of the baseline method turns out in improving the classifier accuracy to 78%.

Information, 2023

Word sense disambiguation (WSD) is a process used to determine the most appropriate meaning of a word in a given contextual framework, particularly when the word is ambiguous. While WSD has been extensively studied for English, it remains a challenging problem for resource-scarce languages such as Hindi. Therefore, it is crucial to address ambiguity in Hindi to effectively and efficiently utilize it on the web for various applications such as machine translation, information retrieval, etc. The rich linguistic structure of Hindi, characterized by complex morphological variations and syntactic nuances, presents unique challenges in accurately determining the intended sense of a word within a given context. This review paper presents an overview of different approaches employed to resolve the ambiguity of Hindi words, including supervised, unsupervised, and knowledge-based methods. Additionally, the paper discusses applications, identifies open problems, presents conclusions, and suggests future research directions.

Word Sense Disambiguation (WSD) aims to disambiguate the words which have multiple sense in a context automatically. Sense denotes the meaning of a word and the words which have various meanings in a context are referred as ambiguous words. WSD is vital in many important Natural Language Processing tasks like MT, IR, TC, SP etc. This research paper attempts to propose a supervised Machine Learning approach-Decision Tree for Word Sense Disambiguation task in Assamese language. A Decision Tree is decision model flow-chart like tree structure where each internal node denotes a test, each branch represents result of a test and each leaf holds a sense label. J48 a Java implementation of C4.5 decision tree algorithm is taken for experimentation in our case. A few polysemous words with different real occurrences in Assamese text with manual sense annotation was collected as the training and test dataset. DT algorithm produces average F-measure of .611 when 10-fold crossvalidation evaluation was performed on 10 Assamese

The problem underlying this research is to solve word sense disambiguation problem for Urdu language text. Although the problem is well-studied for English language text, the work on Urdu is still in infancy. Since Urdu is a morphologically a rich language and hence very sparse, it may not be guaranteed that the methods that performance better for English language may also be effective for Urdu. The goal of this research is to propose a method for word sense disambiguation of ambiguous Urdu words and develop a prototype system. We have proposed a two-stage method where the first stage is detection phase and second stage is identification phase. The detection task is performed by maintaining a lexicon of ambiguous words. The identification problem is posed as a classification problem where a classifier is trained for each ambiguous word by developing a word sense disambiguation specific test collection. Two well-studied text classification algorithms have been analyzed for classification: support vector machine and Naïve Bayes. The results have shown that support vector machine has an advantage due to it intrinsic ability to handle sparse data and less sensitivity to high dimensionality of dataset. I. INTROUDCTION Information extraction (IE) is the task of automatically extracting structured information from unstructured and/or semi-structured machine-readable documents.. This extraction is basically applied on structured and semi-structured textual data that is readable to machine. However, the techniques of automatic annotation and content extraction from multimedia are also considered in IE. In IE, main goal is to automatically extract entities from data on the web, finding the relationships among entities and describing attributes of entities. The extracted knowledge enables much richer forms of queries on the abundant unstructured sources than possible with keyword searches alone. Another objective of IE is to represent the unstructured text in a form useful to derive inferences that are based on the logical content found in the input data. Assigning a correct sense automatically to a word that has more than one meaning is referred as word sense disambiguation (WSD). Many words have more than one meaning having correct sense in a particular context for which WSD technique is introduced. Word sense disambiguation is a process to resolve the ambiguity and rectify the use of word in a specific frame of reference [1]. Human are good to identify the correct sense of word with the help of context easily. WSD is a domain that deals with modeling this ability of humans into computers. Frequently it is noticed that some words may have multiple meanings according to their use as parts Of speech (POS). In the following example same word (bear) gives us a sense of verb and noun in two different sentences.

2009

Abstract Word sense disambiguation (WSD) is the ability to identify the meaning of words in context in a computational manner. WSD is considered an AI-complete problem, that is, a task whose solution is at least as hard as the most difficult problems in artificial intelligence. We introduce the reader to the motivations for solving the ambiguity of words and provide a description of the task. We overview supervised, unsupervised, and knowledge-based approaches.

Sādhanā, 2019

In this work, Word Sense Disambiguation (WSD) in Bengali language is implemented using unsupervised methodology. In the first phase of this experiment, sentence clustering is performed using Maximum Entropy method and the clusters are labelled with their innate senses by manual intervention, as these sense-tagged clusters could be used as sense inventories for further experiment. In the next phase, when a test data comes to be disambiguated, the Cosine Similarity Measure is used to find the closeness of that test data with the initially sense-tagged clusters. The minimum distance of that test data from a particular sense-tagged cluster assigns the same sense to the test data as that of the cluster it is assigned with. This strategy is considered as the baseline strategy, which produces 35% accurate result in WSD task. Next, two extensions are adopted over this baseline strategy: (a) Principal Component Analysis (PCA) over the feature vector, which produces 52% accuracy in WSD task and (b) Context Expansion of the sentences using Bengali WordNet coupled with PCA, which produces 61% accuracy in WSD task. The data sets that are used in this work are obtained from the Bengali corpus, developed under the Technology Development for the Indian Languages (TDIL) project of the Government of India, and the lexical knowledge base (i.e., the Bengali WordNet) used in the work is developed at the Indian Statistical Institute, Kolkata, under the Indradhanush Project of the DeitY, Government of India. The challenges and the pitfalls of this work are also described in detail in the pre-conclusion section.

The task of assigning proper meaning to an ambiguous word in a particular context is termed word sense disambiguation (WSD). We propose a genetic algorithm, improved by local search techniques, to maximise the overall semantic similarity or relatedness of a given text. Local search is used because of the inefficiency of populationbased algorithms (e.g. genetic algorithm) in exploiting the search space. Firstly, the proposed method assigns all potential senses for each word using a WordNet sense inventory. Then, the improved genetic algorithm is applied to determine a coherent set of senses that carries maximum similarity or relatedness score based on information content and gloss overlap methods, namely extended Lesk algorithm and Jiang and Conrath (jcn). The obtained results outperformed other unsupervised methods, which are related to the proposed method, when tested on the same benchmark dataset. It can be concluded that the proposed method is an effective solution for unsupervised WSD.

Sadhana (2019)44:181, 2019

An attempt is made in this paper to report how a supervised methodology has been adopted for the task of Word Sense Disambiguation (WSD) in Bengali with necessary modifications. At the initial stage, four commonly used supervised methods, Decision Tree (DT), Support Vector Machine (SVM), Artificial Neural Network (ANN) and Naı¨veNaı¨ve Bayes (NB), are developed at the baseline. These algorithms are applied individually on a data set of 13 most frequently used Bengali ambiguous words. On experimental basis, the baseline strategy is modified with two extensions: (a) inclusion of lemmatization process into the system and (b) bootstrapping of the operational process. As a result, the levels of accuracy of the baseline methods are slightly improved, which is a positive signal for the whole process of disambiguation as it opens scope for further modification of the existing method for better result. In this experiment, the data sets are prepared from the Bengali corpus, developed in the Technology Development for Indian Languages (TDIL) project of the Government of India and from the Bengali WordNet, which is developed at the Indian Statistical Institute, Kolkata. The paper reports the challenges and pitfalls of the work that have been closely observed during the experiment.

Word Sense Disambiguation (WSD) is the task of identifying the correct sense of a word in a specific context when the word has multiple meaning. WSD is very important as an intermediate step in many Natural Language Processing (NLP) tasks, especially in Information Extraction(IE), Machine Translation(MT) and Question/Answering Systems. Word sense ambiguity arises when a particular word has more than one possible sense. The peculiarity of any language is that it includes a lot of ambiguous words. Since the sense of a word depends on its context of use, disambiguation process requires the understanding of word knowledge. Automatic WSD systems are available for structured languages like English, Chinese, etc. But Indian languages are morphologically rich and thus the processing task is very complex. The aim of this work is to develop a WSD system for Malayalam, a language spoken in India, predominantly used in the state of Kerala. The proposed system uses a corpus which is collected from various Malayalam web documents. For each possible sense of the ambiguous word, a relatively small set of training examples (seed sets) are identified which represents the sense. Collocations and most co-occurring words are considered as training examples. Seed set expansion module extends the seed set by adding most similar words to the seed set elements. These extended sets act as sense clusters. The most similar sense cluster to the input text context is considered as the sense of the target word.