Linear discriminant analysis in document classification

2006

Many studies in automated Text Categorization focus on the performance of classifiers, with or without considering feature selection methods, but almost as a rule taking into account just one document representation. Only relatively recently did detailed studies on the impact of various document representations step into the spotlight, showing that there may be statistically significant differences in classifier performance even among variations of the classical bag-of-words model. This paper examines the relationship between the idf transform and several widely used feature selection methods, in the context of Naïve Bayes and Support Vector Machines classifiers, on datasets extracted from the dmoz ontology of Web-page descriptions. The described experimental study shows that the idf transform considerably effects the distribution of classification performance over feature selection reduction rates, and offers an evaluation method which permits the discovery of relationships between different document representations and feature selection methods which is independent of absolute differences in classification performance.

Hybrid Information Systems, A. Abraham and …, 2002

Linear Discriminant (LD) techniques are typically used in pattern recognition tasks when there are many (n >> 10 4 ) datapoints in low-dimensional (d < 10 2 ) space. In this paper we argue on theoretical grounds that LD is in fact more appropriate when training data is sparse, and the dimension of the space is extremely high. To support this conclusion we present experimental results on a medical text classification problem of great practical importance, autocoding of adverse event reports. We trained and tested LD-based systems for a variety of classification schemes widely used in the clinical drug trial process (COSTART, WHOART, HARTS, and MedDRA) and obtained significant reduction in the rate of misclassification compared both to generic Bayesian machine-learning techniques and to the current generation of domain-specific autocoders based on string matching.

2008

Text categorization is an important research area and has been receiving much attention due to the growth of the on-line information and of Internet. Automated text categorization is generally cast as a multi-class classification problem. Much of previous work focused on binary document classification problems. Support vector machines (SVMs) excel in binary classification, but the elegant theory behind large-margin hyperplane cannot be easily extended to multi-class text classification.

Automatic text classification (ATC) is the task of automatically assigning a set of documents into appropriate categories (or classes, or topics). One of the feature generation techniques is extracting absolute word frequency from textual documents to be used as feature vectors in machine learning techniques. One of the limitations of this technique is the dependency on text length leading into lower classification rates. Another problem in ATC is the high dimensional space. We present a performance evaluation of feature transformation techniques and regularized linear discriminant function (RLD) in automatic text classification. Moreover we provide experimental evaluation of Principal Component Analysis (PCA) in reducing the high dimensionality. Feature transformation techniques used considerably improved the classification accuracy, and RLD outperformed all classifiers used. Experimental results showed effective dimension reduction.

International Journal of Data Mining & Knowledge Management Process, 2012

In this paper we propose a new method of classifying text documents. Unlike conventional vector space models, the proposed method preserves the sequence of term occurrence in a document. The term sequence is effectively preserved with the help of a novel datastructure called 'Status Matrix'. Further the corresponding classification technique has been proposed for efficient classification of text documents. In addition, in order to avoid sequential matching during classification, we propose to index the terms in Btree, an efficient index scheme. Each term in B-tree is associated with a list of class labels of those documents which contain the term. Further the corresponding classification technique has been proposed. To corroborate the efficacy of the proposed representation and status matrix based classification, we have conducted extensive experiments on various datasets.

BRDC Fearture Ranking in Text Classificaiton Paper_66-Towards_Dimension_Reduction in Filter Based Text Classification, 2024

The volume and complexity of textual data have significantly increased worldwide, demanding a comprehensive understanding of machine learning techniques for accurate text classification in various applications. In recent years, there has been significant growth in natural language processing (NLP) and neural networks (NNs). Deep learning (DL) models have outperformed classical machine learning approaches in text classification tasks, such as sentiment analysis, news categorization, question answering, and natural language inference. Dimension reduction is crucial for refining the classifier performance and decreasing the computational cost of text classification. Existing methodologies, such as the Improved Relative Discrimination Criterion (IRDC) and the Relative Discrimination Criterion (RDC), exhibit deficiencies in proper normalization and are not well-balanced regarding distinct class's term ranking. This study introduced an improved feature-ranking metric called the Balanced Relative Discrimination Criterion (BRDC). This study measured document frequencies into termcount estimations, facilitating a normalized and balanced classification approach. The proposed methodology demonstrated superior performance compared to existing techniques. Experiments were conducted to evaluate the efficacy of the proposed techniques using Decision Tree (DT), Logistic Regression (LR), Multinomial Naïve Bayes (MNB), and Long Short-Term Memory (LSTM) models on three benchmark datasets: Reuters-21578, 20newsgroup, and AG News. The findings indicate that LSTM outperformed the other models and can be applied in conjunction with the proposed BRDC approach.

2011

The curse of dimensionality is a well-recognized problem in the field of document filtering. In particular, this concerns methods where vector space models are utilized to describe the document-concept space. When performing content classification across a variety of topics, the number of different concepts (dimensions) rapidly explodes and as a result many techniques are rendered inapplicable. Furthermore the extent of information represented by each of the concepts may vary significantly. In this paper, we present a dimensionality reduction approach which approximates the user's preferences in the form of value function and leads to a quick and efficient filtering procedure. The proposed system requires the user to provide preference information in the form of a training set in order to generate a search rule. Each document in the training set is profiled into a vector of concepts. The document profiling is accomplished by utilizing Wikipedia-articles to define the semantic information contained in words which allows them to be perceived as concepts. Once the set of concepts contained in the training set is known, a modified Wilks' lambda approach is used for dimensionality reduction by ensuring minimal loss of semantic information.

2015

High dimension of bag-of-words vectors poses a serious challenge from sparse data, overfitting, irrelevant features to document classification. Filter feature selection is one of effective methods for dimensionality reduction by removing irrelevant features from feature set. This paper focuses on two main problems of filter feature selection which are the feature score computation and the imbalance in the feature selection performance between categories. We propose a novel filter feature selection method, named ExFCFS, to comprehensively resolve these problems. We experiment on related filter feature selection methods with two benchmark datasets Reuters-21578 dataset and Ohsumed dataset. The experimental results show the effectiveness of our solutions in terms of both Micro-F1 measure and Macro-F1 measure. Keywords— bag-of-words vector, filter feature selection, document classification

IEICE Transactions on Information and Systems, 2008

Feature transformation in automatic text classification (ATC) can lead to better classification performance. Furthermore dimensionality reduction is important in ATC. Hence, feature transformation and dimensionality reduction are performed to obtain lower computational costs with improved classification performance. However, feature transformation and dimension reduction techniques have been conventionally considered in isolation. In such cases classification performance can be lower than when integrated. Therefore, we propose an integrated feature analysis approach which improves the classification performance at lower dimensionality. Moreover, we propose a multiple feature integration technique which also improves classification effectiveness. key words: text classification/categorization, feature transformation, dimension reduction, principal component analysis, canonical discriminant analysis, integrated feature analysis, multiple feature integration 2. Integrated Feature Analysis (IFA) This section describes techniques on which the proposed approach is based. We name this framework the integrated feature analysis (IFA). This includes the descriptions of the multiple feature integration (MFI).

2006

In this paper we present a multiclassifier approach for multilabel document classification problems, where a set of k-NN classifiers is used to predict the category of text documents based on different training subsampling databases. These databases are obtained from the original training database by random subsampling. In order to combine the predictions generated by the multiclassifier, Bayesian voting is applied. Through all the classification process, a reduced dimension vector representation obtained by Singular Value Decomposition (SVD) is used for training and testing documents. The good results of our experiments give an indication of the potentiality of the proposed approach.