Probabilistic Explicit Topic Modeling

I present an in-detail introduction to Topic Models (TM), a family of probabilistic models for (mainly) document modeling. I introduce and motivate the model, and illustrate its applications in Natural Language Processing (NLP), with the particular focus on a thorough description and derivation of the common inference algorithms proposed for TMs. I also compare the algorithms, overviewing various evaluation methods, and provide practical suggestions. Finally I look at a few popular extensions of TM before concluding.

I present here an overview of recent advances in probabilistic topic modelling and related Bayesian graphical models as well as some of their more atypical applications outside of their home: text analysis. These techniques allow the modelling of high dimensional count vectors with strong correlations. With such data, simply calculating a correlation matrix is infeasible. Probabilistic topic models address this using mixtures of multinomials estimated via Bayesian inference with Dirichlet priors. The use of conjugate priors allows for efficient inference, and these techniques scale well to data sets with many millions of vectors. The first of these techniques to attract significant attention was Latent Dirichlet Allocation (LDA) [1, 2]. Numerous extensions and adaptations of LDA have been proposed: non-parametric models; assorted models incorporating authors, sentiment and other features; models regularised through the use of extra metadata or extra priors on topic structure, and many more [3]. They have become widely used in the text analysis and population genetics communities, with a number of compelling applications. These techniques are not restricted to text analysis, however, and can be applied to other types of data which can be sensibly discretised and represented as counts of labels/properties/etc. LDA and it's variants have been used to find patterns in data from diverse areas of inquiry, including genetics, plant physiology, image analysis, social network analysis, remote sensing and astrophysics. Nonetheless, it is relatively recently that probabilistic topic models have found applications outside of text analysis, and to date few such applications have been considered. I suggest that there is substantial untapped potential for topic models and models inspired by or incorporating topic models to be fruitfully applied, and outline the characteristics of systems and data for which this may be the case.

Journal of Computer Science

This paper introduces the Poisson-Gamma Latent Dirichlet Allocation (PGLDA) model for modeling word dependencies in topics modeling. The Poisson document length distribution has been used extensively in the past for modeling topics with the expectation that its effect will fizzle out at the end of the model definition. This procedure often leads to downplaying the effect of word correlation with topics and thus reducing the precision or accuracy of retrieved documents in such a situation. Therefore, we propose a new class of model that relaxes the words independence assumption in the existing Latent Dirichlet Allocation (LDA) model by introducing the Gamma distribution that can capture the correlation between adjacent words in a document. The Poisson document length distribution and Gamma correlation distribution are then convoluted to form a new mixture distribution for modeling word dependencies. Model parameter estimation was achieved via Laplacian approximation of the log-likelihood. The new model was then evaluated using the 20 Newsgroups and AG's News datasets. The applicability of the model was assessed using the F1 score. The results of the evaluation showed appreciable supremacy of PGLDA over LDA.

Proceedings Book – AI 2024 AI Horizon - A new era of creativity and innovation, 2024

Latent Dirichlet Allocation, also known as LDA, is a generative probabilistic model in the domain of Natural Language Processing (NLP), that is frequently used for the purpose of discovering latent thematic structures that are concealed inside large text datasets. The first part of the paper is a Literature Review of LDA Applied in NLP. This review addresses LDA's most important improvements to the NLP field and demonstrates how well it works at finding semantic structures in unstructured text. Next, the paper discusses Parameter Estimation Methods. It covers methods like Variational Inference and Gibbs Sampling that are used to guess how topics are spread out across texts and words. These methods are tested to see how well they work with big datasets and how quickly they can be computed. In the Hyperparameter Tuning section, we explore what happens when we change important hyperparameters like the Dirichlet priors: topic-document distribution and word-topic distribution. We also discuss ways to improve model performance, such as grid search and Bayesian optimization. Finally, the paper addresses the limitations of LDA, such as the assumption of "bag-of-words," the fact that topics can only be roughly understood, the use of "Dirichlet priors," the ability to scale, and the sensitivity to hyperparameters. The purpose of this work is to give researchers and practitioners with a comprehensive understanding of the statistical mechanics of LDA and to guide its ap plication in a variety of domains.

Nigerian Journal of Technology, 2019

In this study, empirical prior Dirichlet allocation (epLDA) model that uses latent semantic indexing framework to derive the priors required for topics computation from data is presented. The parameters of the priors so obtained are related to the parameters of the conventional LDA model using exponential function. The model was implemented and tested with benchmarked data and it achieves a prediction accuracy of 92.15%. It was observed that the epLDA model consistently outperforms the conventional LDA model on different datasets with an average percentage accuracy of 6.33%; this clearly demonstrates the advantage of using side information obtained from data for the computation of the mixture components. Keywords: latent Dirichlet allocation; semantic indexing; empirical prior; hidden structures; Prediction accuracy

International Journal of Advanced Computer Science and Applications, 2019

The earliest modification of Latent Dirichlet Allocation (LDA) in terms of words or document attributes is by relaxing its exchangeability assumption via the Bag-of-word (BoW) matrix. Several authors have proposed many modifications of the original LDA by focusing on model that assumes the current topic depends on the words from previous topic. Most of the earlier work ignored the document length distribution since it is assumed that it will fizzle out at the modelling stage. Thus, in this paper, the Poisson document length distribution of LDA model is replaced with Generalized Poisson (GP) distribution which has the strength of capturing complex structures. The main strengths of GP are in capturing overdispersed (variance larger than mean) and under dispersed (variance smaller than mean) count data. The Poisson distribution used by LDA strongly relies on the assumption that the mean and variance of document lengths are equal. This assumption is often unrealistic with most real-life text data where the variance of document length may be greater than or less than their mean. Approximate estimate of the GPLDA model parameters was achieved using Newton-Raphson approximation technique of log-likelihood. Performance and comparative analysis of GPLDA with LDA using accuracy and F 1 showed improved results.

International Journal of Information and Communication Technology Research, 2017

Probabilistic topic models have been very popular in automatic text analysis since their introduction. These models work based on word co-occurrence, but are not very flexible with respect to the context in which co-occurrence is considered. Many probabilistic topic models do not allow for taking local or spatial data into account. In this paper, we introduce a probabilistic topic model that benefits from an arbitrary-length co-occurrence window and encodes local word dependencies for extracting topics. We assume a multinomial distribution with Dirichlet prior over the window positions to let the words in every position have a chance to influence topic assignments. In the proposed model, topics being shown by word pairs have a more meaningful presentation. The model is applied on a dataset of 2000 documents. The proposed model produces interesting meaningful topics and reduces the problem of sparseness.

2018

Machine Learning, 2013

Probabilistic topic models are widely used in different contexts to uncover the hidden structure in large text corpora. One of the main (and perhaps strong) assumption of these models is that generative process follows a bag-of-words assumption, i.e each token is independent from the previous one. We extend the popular Latent Dirichlet Allocation model by exploiting three different conditional Markovian assumptions: (i) the token generation depends on the current topic and on the previous token; (ii) the topic associated with each observation depends on topic associated with the previous one; (iii) the token generation depends on the current and previous topic. For each of these modeling assumptions we present a Gibbs Sampling procedure for parameter estimation. Experimental evaluation over real-word data shows the performance advantages, in terms of recall and precision, of the sequence-modeling approaches.

ICST Transactions on Scalable Information Systems

Topic modelling is the new revolution in text mining. It is a statistical technique for revealing the underlying semantic structure in large collection of documents. After analysing approximately 300 research articles on topic modeling, a comprehensive survey on topic modelling has been presented in this paper. It includes classification hierarchy, Topic modelling methods, Posterior Inference techniques, different evolution models of latent Dirichlet allocation (LDA) and its applications in different areas of technology including Scientific Literature, Bioinformatics, Software Engineering and analysing social network is presented. Quantitative evaluation of topic modeling techniques is also presented in detail for better understanding the concept of topic modeling. At the end paper is concluded with detailed discussion on challenges of topic modelling, which will definitely give researchers an insight for good research.