Latent Dirichlet allocation

Latent Dirichlet allocation (LDA) is a generative probabilistic model for collections of discrete observations. It represents each document as a mixture of latent topics and each topic as a probability distribution over a fixed vocabulary. Although the terminology refers to documents and words, the same mathematical structure applies whenever grouped observations exhibit recurring patterns of co-occurrence.

LDA belongs to the family of Bayesian hierarchical models. Its name identifies three central features. The topics are latent because they are not directly observed. The model uses Dirichlet distributions as priors over categorical probability vectors. Allocation refers to the assignment of each observed word token to a latent topic.

The model was introduced in its standard form in 2003 by David M. Blei, Andrew Y. Ng, Michael I. Jordan, and You Watanabe. Their formulation combined a document-level mixture model with topic-specific word distributions and supplied a variational method for approximating the resulting posterior distribution. It subsequently became a standard reference point for probabilistic approaches to topic modeling.

Probabilistic formulation

Let a corpus contain (D) documents and let the vocabulary contain (V) distinct word types. The model assumes (K) latent topics. Topic (k) is represented by a probability vector

[ \boldsymbol{\beta}_k

(\beta_{k1},\ldots,\beta_{kV}), ]

where (\beta_{kv}) is the probability of vocabulary term (v) under topic (k), subject to

[ \sum_{v=1}^{V}\beta_{kv}=1. ]

Each document (d) has its own topic-proportion vector

[ \boldsymbol{\theta}_d

(\theta_{d1},\ldots,\theta_{dK}), ]

whose components are nonnegative and sum to one. The value (\theta_{dk}) expresses the probability that a word token in document (d) is associated with topic (k).

In the classical formulation, the document-specific vector follows a Dirichlet distribution,

[ \boldsymbol{\theta}_d \sim \operatorname{Dirichlet}(\boldsymbol{\alpha}), ]

where the hyperparameter (\boldsymbol{\alpha}) determines the prior concentration and relative prevalence of the topics. Some formulations treat the topic-word vectors as fixed model parameters. Fully Bayesian variants instead assign them a second Dirichlet prior,

[ \boldsymbol{\beta}_k \sim \operatorname{Dirichlet}(\boldsymbol{\eta}). ]

For the (n)-th token in document (d), the model draws a topic assignment

[ z_{dn} \sim \operatorname{Categorical}(\boldsymbol{\theta}_d), ]

followed by an observed word

[ w_{dn} \sim \operatorname{Categorical}(\boldsymbol{\beta}{z{dn}}). ]

The resulting joint distribution, with topic-word distributions treated as random variables, is

[ p(\boldsymbol{\beta},\boldsymbol{\theta}, \mathbf{z},\mathbf{w}\mid \boldsymbol{\alpha},\boldsymbol{\eta})

\prod_{k=1}^{K} p(\boldsymbol{\beta}k\mid\boldsymbol{\eta}) \prod{d=1}^{D} p(\boldsymbol{\theta}d\mid\boldsymbol{\alpha}) \prod{n=1}^{N_d} p(z_{dn}\mid\boldsymbol{\theta}d) p(w{dn}\mid z_{dn},\boldsymbol{\beta}), ]

where (N_d) denotes the number of observed tokens in document (d).

Exchangeability and representation

LDA uses the bag-of-words model, under which token order is absent from the likelihood. Words remain grouped by document, but permutations within a document have identical probability. Documents are likewise exchangeable when no higher-level corpus structure is included.

This exchangeability assumption connects LDA with the de Finetti theorem. A document is represented as a conditionally independent sequence once its topic proportions and token-level topic assignments are given. The marginal dependence among its words arises from their shared document-level mixture.

The representation differs from a finite mixture of documents. In an ordinary mixture model, an entire document is associated with a single latent component. In LDA, each token receives its own topic assignment, while all assignments in the same document depend on a common topic-proportion vector. A document may therefore exhibit several recurring co-occurrence structures without being partitioned into separate documents.

Topics are statistical distributions rather than semantic labels. Human-readable interpretations arise from words receiving relatively high probability under a topic, but semantic coherence is not an explicit random variable in the model. The indices assigned to topics also have no intrinsic meaning because any simultaneous permutation of the topic labels leaves the likelihood unchanged. This symmetry is an instance of label switching.

Posterior inference

The main inferential quantity is the posterior distribution

[ p(\boldsymbol{\theta},\mathbf{z},\boldsymbol{\beta} \mid \mathbf{w},\boldsymbol{\alpha},\boldsymbol{\eta}). ]

Exact computation is generally intractable because evaluation requires summation over the possible topic assignment of every token and integration over continuous probability vectors. Practical implementations therefore use approximate inference.

The original treatment developed by Blei, Ng, Jordan, and Watanabe used variational inference. That method introduces a tractable family of distributions over the latent variables and minimizes the Kullback–Leibler divergence between the approximation and the posterior. Equivalently, optimization maximizes an evidence lower bound on the logarithm of the corpus probability.

A common mean-field approximation factorizes as

[ q(\boldsymbol{\theta}_d,\mathbf{z}_d)

q(\boldsymbol{\theta}d\mid\boldsymbol{\gamma}d) \prod{n=1}^{N_d} q(z{dn}\mid\boldsymbol{\phi}_{dn}), ]

where (\boldsymbol{\gamma}d) parameterizes an approximate Dirichlet distribution and (\boldsymbol{\phi}{dn}) parameterizes an approximate categorical distribution. Coordinate updates alternate between document-level topic proportions and token-level assignment probabilities until the variational objective reaches a stationary point.

Thomas L. Griffiths and Mark Steyvers developed an influential alternative based on Gibbs sampling. In the collapsed form of this algorithm, the document-topic and topic-word probability vectors are integrated out analytically through Dirichlet–categorical conjugacy. Sampling then operates directly on the token-level topic assignments.

For a token whose observed vocabulary index is (v), the collapsed conditional probability is proportional to

[ p(z_{dn}=k\mid \mathbf{z}{-dn},\mathbf{w}) \propto \frac{n{dk}^{-dn}+\alpha_k} {n_d^{-dn}+\sum_j\alpha_j} , \frac{n_{kv}^{-dn}+\eta_v} {n_k^{-dn}+\sum_u\eta_u}. ]

Here (n_{dk}^{-dn}) counts the other tokens in document (d) assigned to topic (k). The quantity (n_{kv}^{-dn}) counts other occurrences of word type (v) assigned to that topic. Superscript (-dn) indicates that the current token is excluded from the relevant counts.

Expectation propagation, sequential Monte Carlo, and stochastic variants of variational inference provide further approximations. Matthew D. Hoffman, David M. Blei, and Francis Bach formulated an online variational method that replaces full-corpus parameter updates with noisy updates derived from subsets of documents. This modification changes the computational organization of inference without changing the underlying LDA generative model.

Parameter estimation and evaluation

Model fitting usually estimates the topic-word distributions together with posterior document representations. In variational expectation–maximization, an approximate posterior over local latent variables forms the expectation step, while corpus-level parameters are updated in the maximization step. In a fully Bayesian treatment, posterior sampling or variational distributions also represent uncertainty in the topic-word probabilities.

The number of topics (K) is fixed in ordinary LDA and is therefore part of the model specification rather than an inferred latent quantity. Different values alter both the granularity of the representation and the dimension of the parameter space. Nonparametric relatives based on the hierarchical Dirichlet process replace this fixed dimension with a prior over a countably unbounded collection of components.

Held-out likelihood and perplexity measure predictive fit to unseen observations. For a held-out corpus containing (N) tokens, perplexity is commonly written as

[ \operatorname{perplexity}

\exp\left( -\frac{\log p(\mathbf{w}_{\mathrm{held\text{-}out}})}{N} \right). ]

A lower value corresponds to a greater average predictive probability under the fitted model. Exact held-out probability remains intractable, so its evaluation also depends on an approximation to posterior inference.

Predictive fit is distinct from interpretability. A model may assign high probability to unseen text while producing topics whose highest-probability words do not form readily identifiable semantic groupings. Topic coherence statistics quantify selected patterns of co-occurrence among prominent words, but they evaluate a property external to the original likelihood.

Statistical structure and limitations

The Dirichlet prior constrains dependence among topic proportions through a specific covariance structure. Increasing one component necessarily reduces the total probability available to the others, while the concentration parameters determine how strongly document mixtures cluster near the center or boundaries of the simplex. The prior does not express arbitrary correlations in which particular topics systematically rise or fall together.

The correlated topic model, introduced by David M. Blei and John D. Lafferty, replaces the Dirichlet distribution with a logistic-normal construction. This substitution permits a more general covariance structure among transformed topic proportions. It also removes the elementary conjugacy that supports several standard LDA inference formulas.

Because ordinary LDA ignores word order, it represents documents with identical word counts identically even when their syntax differs. The model also assumes that every occurrence is generated from the same corpus-wide topic-word distributions. Temporal change, author-specific usage, and document linkage require additional latent structure rather than different interpretations of the base variables.

Vocabulary construction affects the observations supplied to the model. Distinct inflected forms occupy separate dimensions unless they have already been combined, while polysemous words remain single observed categories despite their context-dependent meanings. Topic assignments may separate some uses through surrounding co-occurrence patterns, but the observed vocabulary symbol itself is unchanged.

LDA is non-identifiable under permutations of its topic indices, and more substantial ambiguities may arise when several parameter settings induce similar distributions over documents. Posterior uncertainty therefore concerns not only individual token assignments but also the decomposition of corpus-level co-occurrence into topics. Point estimates suppress part of this uncertainty even when they provide a compact representation.

Relation to matrix factorization

LDA has a close conceptual relationship with probabilistic latent semantic analysis (pLSA), developed by Thomas Hofmann. Both models express a document as a mixture over latent topics and express each topic as a distribution over words. Their treatment of document mixtures differs.

In pLSA, each training document has its own parameter vector without a generative prior that applies to new documents. LDA places a Dirichlet distribution over document-level mixtures, producing a probability model for previously unseen documents. Inference for a new document estimates its latent topic proportions while retaining the corpus-level topic distributions.

The expected word-frequency vector of document (d) satisfies

[ \mathbb{E}[\mathbf{x}_d] \propto \boldsymbol{\theta}_d^\mathsf{T}\mathbf{B}, ]

where the rows of (\mathbf{B}) contain the topic-word distributions. This expression resembles non-negative matrix factorization, but LDA additionally specifies token-level sampling, prior distributions, and posterior uncertainty. The resemblance concerns the low-dimensional decomposition rather than identity between the statistical objectives.

Applications

LDA has been used to summarize recurring co-occurrence patterns in scientific literature, archival collections, and other large text corpora. A fitted document-topic representation also supplies lower-dimensional variables for downstream statistical analysis. In such uses, the inferred topics remain properties of the fitted probabilistic model rather than externally verified categories.

The same construction extends beyond lexical data when observations are discrete and organized into groups. A document may correspond to an individual record, while words may correspond to coded events associated with that record. The mathematical interpretation remains a mixed-membership model in which each group has a distribution over latent components and each observation has a component-specific allocation.