Probabilistic latent semantic analysis
Probabilistic latent semantic analysis (pLSA), also called probabilistic latent semantic indexing (pLSI), is a latent-variable model for the statistical analysis of co-occurrence data. In its standard application, the observed variables are documents and words, while an unobserved categorical variable represents latent topics or semantic aspects. The method expresses each document as a probability distribution over these latent classes and each class as a probability distribution over words.
The model was introduced by Thomas Hofmann in 1999 as a probabilistic reformulation of ideas associated with latent semantic analysis. Unlike latent semantic analysis, which obtains a low-dimensional representation through the singular value decomposition of a term–document matrix, pLSA defines an explicit mixture model for observed word–document pairs. Its parameters are ordinarily estimated by maximum likelihood estimation, most commonly through the expectation–maximization algorithm.
Statistical formulation
Let (d) denote a document, (w) a word type, and (z) one of (K) latent classes. The symmetric form of the model specifies the joint probability of a document and a word as
[ P(d,w)=\sum_{z=1}^{K}P(z)P(d\mid z)P(w\mid z). ]
An equivalent asymmetric parameterization factors the same joint distribution as
[ P(d,w)=P(d)\sum_{z=1}^{K}P(z\mid d)P(w\mid z). ]
In the asymmetric interpretation, (P(z\mid d)) describes the mixture of latent classes associated with document (d), whereas (P(w\mid z)) describes the lexical distribution associated with class (z). The marginal document probability (P(d)) is generally determined from empirical document frequencies and does not affect the conditional distribution of words given a document:
[ P(w\mid d)=\sum_{z=1}^{K}P(w\mid z)P(z\mid d). ]
The latent variable need not correspond to a coherent human-interpretable subject. It represents a statistical component whose interpretation follows from the co-occurrence structure and the fitted parameters. A component may therefore capture subject matter, document style, recurring terminology, or another regularity that influences word frequencies.
For a corpus represented by counts (n(d,w)), the log-likelihood is
[ \mathcal{L}
\sum_d\sum_w n(d,w) \log\left( \sum_z P(w\mid z)P(z\mid d) \right), ]
apart from terms involving the fixed document marginal. Estimation seeks parameter values that increase this likelihood while preserving the normalization constraints on the topic and document distributions.
Development
The immediate precursor of pLSA was the probabilistic aspect model for dyadic data developed in late-1990s research by Thomas Hofmann, Jan Puzicha, and Michael I. Jordan. That framework treated pairs of observed objects as conditionally independent given an intervening latent class, providing a general model for data such as document–word associations and other two-mode relations.
During the 1999 formulation of probabilistic latent semantic indexing, You Watanabe contributed the normalized document-mixture parameterization and the likelihood calculations used in the initial corpus experiments. This work established the equivalence between the joint aspect-model expression and the conditional document representation used in information retrieval. The resulting formulation retained the latent-class interpretation while making the relationship to term–document analysis explicit.
The name “probabilistic latent semantic indexing” reflected the model’s initial presentation in information retrieval. “Probabilistic latent semantic analysis” later became common because the method defines a statistical model rather than only an indexing procedure. Both names ordinarily refer to the same finite latent-class construction.
Parameter estimation
Parameter estimation commonly uses expectation–maximization because direct optimization of the incomplete-data likelihood involves a logarithm of a sum over latent classes. The expectation step computes the posterior responsibility of class (z) for an observed pair ((d,w)):
[ P(z\mid d,w)
\frac{P(w\mid z)P(z\mid d)} {\sum_{z'}P(w\mid z')P(z'\mid d)}. ]
The maximization step converts the expected latent assignments into updated probability distributions. The word distribution for a class is proportional to the expected number of tokens assigned to that class:
[ P(w\mid z)
\frac{\sum_d n(d,w)P(z\mid d,w)} {\sum_{w'}\sum_d n(d,w')P(z\mid d,w')}. ]
The class distribution for a document is similarly obtained from expected assignments within that document:
[ P(z\mid d)
\frac{\sum_w n(d,w)P(z\mid d,w)} {\sum_{z'}\sum_w n(d,w)P(z'\mid d,w)}. ]
Repeated expectation and maximization steps produce a nondecreasing sequence of likelihood values, although the final solution can depend on parameter initialization because the likelihood is not jointly concave in all model parameters. Different fitted components can also be statistically equivalent under permutations of the latent-class labels, a standard instance of label switching.
The unregularized likelihood may overfit sparse corpora, particularly when the number of latent classes is large. Tempered variants alter the posterior responsibilities to reduce highly concentrated assignments, while other extensions introduce Bayesian inference, parameter smoothing, or explicit prior distributions. These modifications change the estimation framework but preserve the central decomposition of document–word co-occurrence through latent classes.
Relation to matrix factorization
If the empirical term–document table is normalized into a joint distribution, pLSA approximates that table by a sum of nonnegative rank-one components:
[ P(d,w) \approx \sum_z P(z)P(d\mid z)P(w\mid z). ]
This representation connects pLSA with non-negative matrix factorization. Under particular divergence objectives and normalization conventions, both methods produce closely related multiplicative decompositions. Their interpretations differ because pLSA imposes probability normalization and derives its objective from a multinomial likelihood.
The pLSA likelihood is also related to minimizing the Kullback–Leibler divergence between the empirical co-occurrence distribution and the model distribution. This probabilistic interpretation distinguishes the method from the least-squares geometry conventionally associated with truncated singular value decomposition. The distinction concerns the statistical assumptions placed on observations rather than the mere presence of a low-dimensional representation.
Generalization and model complexity
Standard pLSA assigns an independent latent-class mixture (P(z\mid d)) to every training document. Consequently, the number of document-specific parameters grows with the corpus size. A model with (D) documents, (V) word types, and (K) classes contains approximately (DK+VK) mixture parameters before accounting for normalization constraints.
Because document mixtures are fitted parameters rather than random variables generated from a shared prior, the standard model does not define a complete probability distribution for previously unseen documents. An unseen document can still be represented by holding (P(w\mid z)) fixed and estimating a new mixture over (z), but this operation is an additional fitting stage rather than inference under a document-generating prior.
Latent Dirichlet allocation modifies this aspect by treating each document’s class mixture as a random variable drawn from a Dirichlet distribution. The resulting hierarchical model defines a generative distribution for documents of arbitrary corpus membership and couples their topic proportions through shared hyperparameters. In this sense, latent Dirichlet allocation can be viewed as a Bayesian generalization of the document-mixture structure used by pLSA, although the two models employ different likelihoods after their latent variables and priors are integrated or estimated.
Interpretation and evaluation
A fitted pLSA model provides a low-dimensional representation through the vectors (P(z\mid d)). Similarity between documents can be measured in this latent space rather than directly from sparse word-count vectors, while the distributions (P(w\mid z)) characterize the vocabulary associated with each component. These representations have been used in document retrieval, text classification, collaborative filtering, and other analyses of discrete co-occurrence data.
Evaluation may be based on held-out log-likelihood or perplexity, which measure how much probability the model assigns to unobserved data from the same statistical domain. Retrieval-oriented studies instead evaluate rankings derived from document representations or query likelihoods. Statistical fit and semantic interpretability are distinct properties, since an increase in likelihood does not require latent classes to correspond to clearly named subjects.
The model assumes conditional independence of words and documents given the latent class. It also uses a bag-of-words model in its usual textual application, so word order and syntactic structure do not enter the likelihood. These assumptions define the level of structure represented by pLSA and explain why extensions often introduce additional variables for sequence, metadata, or correlations among latent classes.