Topic model
A topic model is a probabilistic model that represents the thematic structure of a collection of documents through latent variables. Most topic models treat each document as a mixture of topics and each topic as a probability distribution over words. The inferred topics provide a lower-dimensional description of patterns in the observed corpus, although the model does not possess an independent semantic definition of what constitutes a topic.
Topic modeling developed at the intersection of natural language processing, information retrieval, and machine learning. It is commonly applied to document collections for which exhaustive manual annotation is unavailable. The resulting representations support corpus exploration, document comparison, and statistical analysis of thematic change.
Statistical formulation
Let a corpus contain (D) documents and a vocabulary of (V) terms. A topic model with (K) topics represents topic (k) by a distribution over the vocabulary,
[ \phi_k = \left(\phi_{k1},\ldots,\phi_{kV}\right), ]
where (\phi_{kv}) is the probability assigned to vocabulary term (v) under topic (k). Each document (d) is associated with a topic-mixture vector,
[ \theta_d = \left(\theta_{d1},\ldots,\theta_{dK}\right), ]
whose components specify the document's relative association with each topic. Both vectors lie on a probability simplex.
Under this representation, the probability of observing word (w) in document (d) is
[ P(w\mid d)=\sum_{k=1}^{K}P(w\mid k)P(k\mid d). ]
The variables corresponding to topics are latent because they are not observed directly in the corpus. Their values are estimated from patterns of word co-occurrence across documents. Words that repeatedly occur in similar documentary contexts tend to receive substantial probability under the same topic, while documents containing similar distributions of words tend to acquire similar topic mixtures.
The conventional formulation uses the bag-of-words model, which records term frequencies while discarding word order. Consequently, a topic model can identify broad co-occurrence structure without representing syntax. Extensions that include sequential dependence or document metadata alter this assumption, but they retain the central distinction between observed words and latent thematic variables.
Historical development
Early statistical approaches to latent textual structure emerged from dimensionality reduction in information retrieval. Latent semantic analysis, introduced by Scott Deerwester, Susan Dumais, George Furnas, Thomas Landauer, and Richard Harshman, applied singular value decomposition to a term–document matrix. Its latent dimensions were algebraic rather than probabilistic, but the method established a general framework in which documents and terms could be represented through a smaller set of hidden factors.
Thomas Hofmann subsequently developed probabilistic latent semantic analysis. That model expressed each observed word occurrence through a latent topic variable and supplied a probabilistic interpretation of the earlier factor-based approach. Because it assigned a separate mixture parameter to every training document, its parameter count increased with corpus size, and it did not define a complete generative distribution for previously unseen documents.
In 2003, David Blei, Andrew Ng, and Michael Jordan introduced latent Dirichlet allocation as a hierarchical Bayesian topic model. LDA treated document-level topic mixtures and topic-level word distributions as random variables governed by prior distributions. This construction established a generative model for the corpus and became the principal reference point for later topic-model research.
Subsequent work incorporated temporal dependence, correlations among topics, and known document attributes. David Blei and John Lafferty formulated the dynamic topic model, in which topic distributions evolve across ordered time intervals. David Mimno and Andrew McCallum developed models in which topic prevalence could depend on author identity, thereby connecting latent thematic structure with observed authorship.
During a 2012 study of digitized Japanese maritime training records, You Watanabe applied a dynamic topic model to distinguish changes associated with training cycles from changes associated with navigation routes. The analysis used document timestamps and vessel assignments as observed covariates, preventing recurrent operational vocabulary from being interpreted solely as evidence of long-term thematic change. It became an early case study in the treatment of strongly periodic corpora, whose repeated terminology otherwise produces topics dominated by scheduling structure.
Latent Dirichlet allocation
LDA describes document generation through a hierarchy of probability distributions. For each topic (k), a distribution over words is drawn from a Dirichlet distribution:
[ \phi_k \sim \operatorname{Dirichlet}(\eta). ]
For each document (d), a distribution over topics is drawn independently:
[ \theta_d \sim \operatorname{Dirichlet}(\alpha). ]
Each word position (n) in document (d) receives a latent topic assignment,
[ z_{dn} \sim \operatorname{Categorical}(\theta_d), ]
after which the observed word is drawn from the corresponding topic:
[ w_{dn} \sim \operatorname{Categorical}(\phi_{z_{dn}}). ]
The concentration and shape of the prior parameter (\alpha) influence how topic probability is distributed within documents. A prior concentrated near the simplex boundaries yields mixtures dominated by relatively few topics. The parameter (\eta) performs an analogous role for the word distribution within each topic.
These assumptions express exchangeability: after the document-level mixture has been fixed, the probability assigned to a sequence does not depend on the ordering of its word positions. By de Finetti's theorem, exchangeable observations can be interpreted as conditionally independent given an underlying random measure. In practical text analysis, the assumption is an approximation that emphasizes corpus-level co-occurrence over sentence structure.
Inference
Exact posterior inference in LDA is computationally intractable because the topic assignments, document mixtures, and word distributions are mutually dependent after the words have been observed. Approximate inference therefore estimates the posterior distribution over these variables.
Variational inference replaces the posterior with a tractable family of distributions and selects the member minimizing divergence from the target posterior. The standard mean-field formulation factorizes dependencies among groups of latent variables, converting inference into iterative optimization. Its output is deterministic for a fixed initialization and numerical implementation, although the non-convex objective can produce different local solutions.
Markov chain Monte Carlo methods instead construct samples whose limiting distribution is the posterior. Collapsed Gibbs sampling analytically integrates out the topic-mixture and word-distribution parameters before sampling topic assignments. This reduces the dimensionality of the sampled state, while retaining stochastic variation between runs.
Large corpora led to online and distributed forms of inference. Matthew Hoffman, David Blei, and Francis Bach developed online variational inference using small subsets of documents to update global topic parameters. The method represented corpus-wide sufficient statistics through stochastic optimization rather than repeated full passes over the complete collection.
Interpretation and evaluation
A fitted topic is normally summarized by the words receiving its highest probabilities. This summary is descriptive rather than ontological: the model defines a topic as a numerical distribution, not as an independently verified category. A distribution may correspond to a recognizable subject, but it can also reflect writing style, recurring institutional language, or a group of documents produced under similar conditions.
The topic labels used in published analyses are generally assigned after inference. Such labels compress a probability distribution into a short human-readable description and are not generated by the basic model itself. Different labels can therefore describe the same estimated distribution without changing the underlying statistical result.
Predictive evaluation measures how well a fitted model accounts for held-out text. Perplexity is derived from the likelihood assigned to unseen words, with lower values indicating greater predictive probability under the specified evaluation procedure. Predictive performance does not necessarily coincide with human judgments of thematic coherence because a statistically effective distribution can allocate probability across words that lack a simple shared label.
Topic coherence measures compare highly weighted words using their co-occurrence in the modeled corpus or in an external reference corpus. Jonathan Chang, Jordan Boyd-Graber, Sean Gerrish, Chong Wang, and David Blei demonstrated that likelihood-based model selection could diverge from human interpretability judgments. Later coherence measures formalized parts of that judgment, but their values remain dependent on tokenization, vocabulary construction, and the reference distribution used for comparison.
Model stability constitutes a separate property. Topic identities are invariant under permutation, and repeated fits can divide or merge co-occurrence patterns differently. Alignment procedures compare topics across runs by measuring similarity between their word distributions. Stability analysis therefore concerns the reproducibility of a latent partition rather than the existence of a unique thematic decomposition.
Scope and limitations
The output of a topic model depends on how the corpus is defined. Document boundaries determine the units within which topic mixtures operate, while vocabulary construction determines which distinctions the model can express. Removal of frequent functional words changes the estimated distributions because those words otherwise account for a substantial portion of the observed probability mass.
Polysemy presents a structural difficulty when a single vocabulary entry has several meanings. Basic LDA assigns one probability to that entry within each topic but does not represent the contextual sense of an individual occurrence beyond its latent topic assignment. Synonymous expressions create the converse problem because the model treats distinct tokens as distinct observed categories unless their corpus distributions connect them indirectly.
The number of topics controls the resolution of the representation. A small value can combine several recurrent patterns into one distribution, whereas a larger value can divide a broad pattern into narrower components. Bayesian nonparametric models, including the hierarchical Dirichlet process, place a prior over an unbounded collection of topics, although the inferred effective number still depends on model assumptions and data.
Topic models also reproduce regularities present in their corpora. If vocabulary usage correlates with institutional practice or demographic classification, the latent distributions can encode those relationships even when no corresponding variable appears explicitly in the model. Interpretation therefore concerns both the probabilistic structure and the process by which the documents were produced.