Sequential latent Dirichlet allocation

2010

We consider the problem of inferring and modeling topics in a sequence of documents with known publication dates. A novel topic model is introduced, based on an extension of the hierarchical Dirichlet process (HDP), termed the dynamic HDP. The details of this general framework may take different forms, depending on the specifics of the model. For the examples considered here we examine base measures based on independent multinomial-Dirichlet measures for representation of topic-dependent word counts. It is assumed that each paragraph is associated with a particular topic, and the probability of a given topic being manifested in a specific document evolves dynamically between contiguous time-stamped documents. Natural simplifications of the dHDP allow fully variational Bayesian (VB) inference, of interest for large-scale problems. We demonstrate results and make comparisons to HDP (i.e., without dynamic topic evolution) and latent Dirichlet allocation (LDA), considering a database of NIPS papers as well as the United States presidential State of the Union addresses from 1790 to 2008.

Proceedings Book – AI 2024 AI Horizon - A new era of creativity and innovation, 2024

Latent Dirichlet Allocation, also known as LDA, is a generative probabilistic model in the domain of Natural Language Processing (NLP), that is frequently used for the purpose of discovering latent thematic structures that are concealed inside large text datasets. The first part of the paper is a Literature Review of LDA Applied in NLP. This review addresses LDA's most important improvements to the NLP field and demonstrates how well it works at finding semantic structures in unstructured text. Next, the paper discusses Parameter Estimation Methods. It covers methods like Variational Inference and Gibbs Sampling that are used to guess how topics are spread out across texts and words. These methods are tested to see how well they work with big datasets and how quickly they can be computed. In the Hyperparameter Tuning section, we explore what happens when we change important hyperparameters like the Dirichlet priors: topic-document distribution and word-topic distribution. We also discuss ways to improve model performance, such as grid search and Bayesian optimization. Finally, the paper addresses the limitations of LDA, such as the assumption of "bag-of-words," the fact that topics can only be roughly understood, the use of "Dirichlet priors," the ability to scale, and the sensitivity to hyperparameters. The purpose of this work is to give researchers and practitioners with a comprehensive understanding of the statistical mechanics of LDA and to guide its ap plication in a variety of domains.

2006

Abstract: We develop a nested hierarchical Dirichlet process (nHDP) for hierarchical topic modeling. The nHDP is a generalization of the nested Chinese restaurant process (nCRP) that allows each word to follow its own path to a topic node according to a document-specific distribution on a shared tree. This alleviates the rigid, single-path formulation of the nCRP, allowing a document to more easily express thematic borrowings as a random effect.

IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000

We consider the problem of inferring and modeling topics in a sequence of documents with known publication dates. The documents at a given time are each characterized by a topic, and the topics are drawn from a mixture model. The proposed model infers the change in the topic mixture weights as a function of time. The details of this general framework may take different forms, depending on the specifics of the model. For the examples considered here we examine base measures based on independent multinomial-Dirichlet measures for representation of topic-dependent word counts. The form of the hierarchical model allows efficient variational Bayesian (VB) inference, of interest for large-scale problems. We demonstrate results and make comparisons to the model when the dynamic character is removed, and also compare to latent Dirichlet allocation (LDA) and topics over time (TOT). We consider a database of NIPS papers as well as the United States presidential State of the Union addresses from 1790 to 2008.

Journal of Computer Science

This paper introduces the Poisson-Gamma Latent Dirichlet Allocation (PGLDA) model for modeling word dependencies in topics modeling. The Poisson document length distribution has been used extensively in the past for modeling topics with the expectation that its effect will fizzle out at the end of the model definition. This procedure often leads to downplaying the effect of word correlation with topics and thus reducing the precision or accuracy of retrieved documents in such a situation. Therefore, we propose a new class of model that relaxes the words independence assumption in the existing Latent Dirichlet Allocation (LDA) model by introducing the Gamma distribution that can capture the correlation between adjacent words in a document. The Poisson document length distribution and Gamma correlation distribution are then convoluted to form a new mixture distribution for modeling word dependencies. Model parameter estimation was achieved via Laplacian approximation of the log-likelihood. The new model was then evaluated using the 20 Newsgroups and AG's News datasets. The applicability of the model was assessed using the F1 score. The results of the evaluation showed appreciable supremacy of PGLDA over LDA.

2012

Topic models are increasingly being used for text analysis tasks, often times replacing earlier semantic techniques such as latent semantic analysis. In this paper, we develop a novel adaptive topic model with the ability to adapt topics from both the previous segment and the parent document. For this proposed model, a Gibbs sampler is developed for doing posterior inference. Experimental results show that with topic adaptation, our model significantly improves over existing approaches in terms of perplexity, and is able to uncover clear sequential structure on, for example, Herman

2005

This paper describes nonparametric Bayesian treatments for analyzing records containing occurrences of items. The introduced model retains the strength of previous approaches that explore the latent factors of each record (e.g. topics of documents), and further uncovers the clustering structure of records, which reflects the statistical dependencies of the latent factors. The nonparametric model induced by a Dirichlet process (DP) flexibly adapts model complexity to reveal the clustering structure of the data. To avoid the problems of dealing with infinite dimensions, we further replace the DP prior by a simpler alternative, namely Dirichlet-multinomial allocation (DMA), which maintains the main modelling properties of the DP. Instead of relying on Markov chain Monte Carlo (MCMC) for inference, this paper applies efficient variational inference based on DMA. The proposed approach yields encouraging empirical results on both a toy problem and text data. The results show that the proposed algorithm uncovers not only the latent factors, but also the clustering structure.

Machine Learning, 2013

Probabilistic topic models are widely used in different contexts to uncover the hidden structure in large text corpora. One of the main (and perhaps strong) assumption of these models is that generative process follows a bag-of-words assumption, i.e each token is independent from the previous one. We extend the popular Latent Dirichlet Allocation model by exploiting three different conditional Markovian assumptions: (i) the token generation depends on the current topic and on the previous token; (ii) the topic associated with each observation depends on topic associated with the previous one; (iii) the token generation depends on the current and previous topic. For each of these modeling assumptions we present a Gibbs Sampling procedure for parameter estimation. Experimental evaluation over real-word data shows the performance advantages, in terms of recall and precision, of the sequence-modeling approaches.

ACM Computing Surveys, 2022

We are not able to deal with a mammoth text corpus without summarizing them into a relatively small subset. A computational tool is extremely needed to understand such a gigantic pool of text. Probabilistic Topic Modeling discovers and explains the enormous collection of documents by reducing them in a topical subspace. In this work, we study the background and advancement of topic modeling techniques. We first introduce the preliminaries of the topic modeling techniques and review its extensions and variations, such as topic modeling over various domains, hierarchical topic modeling, word embedded topic models, and topic models in multilingual perspectives. Besides, the research work for topic modeling in a distributed environment, topic visualization approaches also have been explored. We also covered the implementation and evaluation techniques for topic models in brief. Comparison matrices have been shown over the experimental results of the various categories of topic modeling....