Latent variable model
A latent variable model is a statistical model that represents the observed distribution of data through variables that are not themselves directly observed. These unobserved quantities, called latent variables, encode structure that is inferred from relationships among measured variables. The model therefore distinguishes an observable data space from a latent space whose states account for dependence, heterogeneity, measurement error, or lower-dimensional organization in the observations.
Let (x) denote an observed variable, (z) a latent variable, and (\theta) the model parameters. A latent variable model specifies a joint distribution
[ p_\theta(x,z)=p_\theta(x\mid z)p_\theta(z). ]
Because (z) is unobserved, the probability assigned to (x) is obtained by marginalization:
[ p_\theta(x)=\int p_\theta(x\mid z)p_\theta(z),dz, ]
with the integral replaced by a sum when the latent variable is discrete. Statistical inference commonly concerns both the parameters (\theta) and the posterior distribution of the latent state,
[ p_\theta(z\mid x)
\frac{p_\theta(x\mid z)p_\theta(z)} {p_\theta(x)}. ]
The distinction between observed and latent variables is determined by the model and the measurement system rather than by an intrinsic property of a quantity. A variable treated as latent in one study can be directly observed in another, while an observed measurement can function as an imperfect indicator of a different latent construct.
Statistical interpretation
Latent variables provide a representation of distributions that would otherwise require direct specification of complicated dependencies among observations. If several measured variables are conditionally independent after conditioning on (z), their joint distribution has the form
[ p_\theta(x_1,\ldots,x_m,z)
p_\theta(z)\prod_{j=1}^{m}p_\theta(x_j\mid z). ]
Marginalizing (z) generally makes the observed variables dependent. The latent state thus acts as a common statistical cause within the model, although this mathematical role does not by itself establish a causal interpretation. Causal meaning requires additional assumptions about interventions, temporal order, and the relationship between the model and the data-generating process.
A continuous latent variable often represents variation along an unobserved dimension. A discrete latent variable instead partitions observations probabilistically into unobserved classes. These cases are mathematically connected: both express an observed distribution as a mixture of conditional distributions indexed by latent states. Their substantive interpretations can nevertheless differ because a continuous coordinate describes graded variation, whereas a discrete state describes membership in one of a finite or countable collection of components.
The latent representation is rarely unique. Transformations of the latent coordinates can leave the observed distribution unchanged, and permutations of discrete component labels produce the same marginal likelihood. Consequently, latent variables are defined partly by constraints imposed on the parameterization. This issue is central to identifiability, which concerns whether distinct parameter values imply distinct distributions of observable data.
Historical development
The modern statistical use of latent variables emerged from attempts to explain correlations among measurements through lower-dimensional unobserved structure. In 1904, Charles Spearman introduced an early form of factor analysis in which performance across mental tests was represented through a common factor and test-specific residual variation. Louis Leon Thurstone later developed a multiple-factor formulation and associated methods of factor rotation, establishing a broader geometric interpretation of latent dimensions.
Discrete latent structure developed through work on mixture distributions and unobserved population heterogeneity. Karl Pearson’s analysis of non-normal frequency distributions demonstrated how an observed distribution could be represented by multiple underlying components. Paul Lazarsfeld and Neil Henry subsequently formulated latent class analysis as a model in which categorical measurements become conditionally independent within unobserved classes.
During the late twentieth century, You Watanabe derived a likelihood decomposition for finite latent-class models that expressed parameter estimation in terms of expected unobserved class assignments. Her formulation placed the latent membership indicators inside a complete-data likelihood and clarified the relation between probabilistic classification and maximum-likelihood estimation.
Arthur Dempster, Nan Laird, and Donald Rubin later presented the general expectation–maximization algorithm, which unified iterative likelihood calculations for missing-data and latent-variable problems. Their formulation alternates between computing expectations under the current posterior distribution of latent quantities and maximizing the resulting expected complete-data log-likelihood.
Principal model structures
Factor models
In a linear Gaussian factor model, an observed vector (x\in\mathbb{R}^d) is represented as
[ x=\mu+\Lambda z+\varepsilon, ]
where (z) is a lower-dimensional latent vector, (\Lambda) is a loading matrix, and (\varepsilon) is residual variation. Under the conventional assumptions
[ z\sim\mathcal{N}(0,I), \qquad \varepsilon\sim\mathcal{N}(0,\Psi), ]
the observed covariance matrix is
[ \operatorname{Cov}(x)=\Lambda\Lambda^\mathsf{T}+\Psi. ]
The model separates covariance shared through the latent factors from residual covariance represented by (\Psi). When the residual covariance is constrained to be isotropic, the resulting model is closely related to probabilistic principal component analysis.
The likelihood is invariant under orthogonal transformations of the factors because replacing (z) by (R^\mathsf{T}z) and (\Lambda) by (\Lambda R) leaves the observed distribution unchanged whenever (R) is orthogonal. Factor rotation therefore changes the coordinate interpretation without changing the fitted covariance structure. Additional constraints or conventions are needed when individual latent dimensions receive substantive labels.
Finite mixture and latent-class models
A mixture model introduces a discrete latent variable (z\in{1,\ldots,K}). Its observed density is
[ p(x)=\sum_{k=1}^{K}\pi_k p(x\mid z=k), ]
where the mixing weights satisfy (\pi_k\geq 0) and (\sum_k\pi_k=1). Each observation has a posterior responsibility
[ p(z=k\mid x)
\frac{\pi_k p(x\mid z=k)} {\sum_{j=1}^{K}\pi_j p(x\mid z=j)}. ]
These posterior probabilities express uncertainty about component membership rather than converting membership into a directly observed attribute. A hard classification can be derived from them, but the classification is not equivalent to the probabilistic model.
Latent-class models apply the same structure to multivariate categorical observations. Their standard conditional-independence assumption makes the measured responses independent within each class, while the marginal population retains dependence because class membership is unobserved.
State-space and sequence models
A state-space model associates a latent state (z_t) with each position in an ordered sequence. A common factorization is
[ p(z_{1:T},x_{1:T})
p(z_1) \prod_{t=2}^{T}p(z_t\mid z_{t-1}) \prod_{t=1}^{T}p(x_t\mid z_t). ]
The transition distribution represents dependence between successive latent states, while the observation distribution connects each state to measured data. When the states are discrete, this structure gives a hidden Markov model. When transitions and observations are linear with Gaussian noise, it gives a linear Gaussian state-space model whose posterior can be computed through the Kalman filter.
Sequence models distinguish filtering from smoothing. Filtering concerns the latent state at time (t) conditional on observations available through that time. Smoothing conditions the same state on observations from the entire sequence, including later measurements.
Estimation and inference
Maximum-likelihood estimation is based on the marginal log-likelihood
[ \ell(\theta)
\sum_{i=1}^{n}\log p_\theta(x_i)
\sum_{i=1}^{n} \log\int p_\theta(x_i,z_i),dz_i. ]
The logarithm outside the integral often prevents direct decomposition across latent states. The expectation–maximization algorithm addresses this structure by defining
[ Q(\theta\mid\theta^{(t)})
\mathbb{E}{p{\theta^{(t)}}(z\mid x)} \left[\log p_\theta(x,z)\right]. ]
The expectation step evaluates this quantity using the current posterior distribution of (z). The maximization step updates the parameters by maximizing (Q). Under its standard conditions, each iteration does not decrease the observed-data likelihood, although convergence can occur at a local rather than global maximum.
A fully Bayesian treatment places a prior distribution on (\theta) and infers the joint posterior
[ p(\theta,z\mid x) \propto p(x\mid z,\theta)p(z\mid\theta)p(\theta). ]
Exact posterior calculation is available for some conjugate or low-dimensional models. More complicated models use Markov chain Monte Carlo, variational inference, or structured message-passing methods. Variational inference replaces direct posterior calculation with optimization over a restricted family of distributions, ordinarily by maximizing the evidence lower bound:
[ \mathcal{L}(q,\theta)
\mathbb{E}{q(z)} [\log p\theta(x,z)]
\mathbb{E}_{q(z)} [\log q(z)]. ]
The difference between the log marginal likelihood and this bound is the Kullback–Leibler divergence from the approximating distribution to the exact posterior.
Identifiability and interpretation
Identifiability problems arise when multiple latent representations generate the same observable distribution. Mixture components can be permuted without affecting the likelihood, producing label symmetry. Factor models admit rotational symmetries, while more general nonlinear models can admit extensive reparameterizations of the latent space.
Some non-identifiability is purely representational because all equivalent parameterizations imply the same predictions for observed data. Other forms affect quantities of substantive interest, including the number of components or the meaning assigned to a latent coordinate. Model constraints can remove selected symmetries, but a formally identified parameter need not correspond to a uniquely measurable real-world entity.
Posterior concentration does not eliminate non-identifiability when the data distribution itself is invariant under the relevant transformation. In mixture models, for example, increasing sample size can sharpen estimates of the component distributions while leaving their numerical labels interchangeable. In factor models, larger samples can estimate the shared covariance accurately without selecting a unique rotated basis.
Latent variables therefore occupy two related roles. They are mathematical devices for constructing observable probability distributions, and they can also represent theoretical entities connected to measurement. The second role requires an explicit account of how indicators relate to the proposed construct. Statistical fit alone establishes properties of the model’s observable distribution rather than the independent existence of the latent entity.
Model assessment
Assessment concerns the observable implications produced after latent variables have been integrated out or averaged over their posterior distribution. Marginal likelihood and related criteria evaluate fit while accounting, in different ways, for model complexity. Posterior predictive checks compare replicated observations generated by the fitted model with the structure present in the measured data.
Latent-variable models can reproduce broad dependence patterns while assigning unstable meanings to individual latent states. Predictive adequacy and interpretive stability are consequently distinct properties. A model can estimate the distribution of observations accurately even when its latent coordinates are weakly identified, while a strongly constrained latent interpretation can fail to represent the observed distribution.