Variational inference
Variational inference is a class of methods for approximating probability distributions through optimization. It is most often applied when the posterior distribution of a latent-variable model cannot be evaluated or normalized directly. The method replaces exact posterior computation with the search for a tractable distribution that is close to the posterior according to a specified divergence.
The central construction converts an integration problem into an optimization problem. This conversion permits the use of deterministic optimization algorithms, stochastic-gradient methods, and parameterized function approximators. Its accuracy depends on the expressive capacity of the approximating family, while its computational behavior depends on the geometry of the resulting objective.
Mathematical formulation
Let (x) denote observed data and let (z) denote latent variables. A probabilistic model specifies the joint density
[ p(x,z)=p(x\mid z)p(z). ]
Bayesian inference concerns the posterior density
[ p(z\mid x)=\frac{p(x,z)}{p(x)}, ]
where the marginal likelihood is
[ p(x)=\int p(x,z),dz. ]
The integral defining (p(x)) is generally the principal computational obstacle. Variational inference introduces an approximating distribution (q(z)) belonging to a tractable family (\mathcal{Q}). The conventional objective minimizes the reverse Kullback–Leibler divergence,
[ q^\star
\operatorname*{arg,min}{q\in\mathcal{Q}} D{\mathrm{KL}}!\left(q(z),|,p(z\mid x)\right). ]
Direct evaluation of this divergence would require the unknown marginal likelihood. The same optimum is obtained by maximizing the evidence lower bound, commonly abbreviated ELBO:
[ \mathcal{L}(q)
\mathbb{E}_{q(z)}[\log p(x,z)]
\mathbb{E}_{q(z)}[\log q(z)]. ]
The relationship between the ELBO and the marginal likelihood is
[ \log p(x)
\mathcal{L}(q) + D_{\mathrm{KL}}!\left(q(z),|,p(z\mid x)\right). ]
Since the divergence is nonnegative, (\mathcal{L}(q)) is a lower bound on (\log p(x)). The bound becomes exact precisely when (q(z)) equals the posterior almost everywhere. When (\mathcal{Q}) excludes the exact posterior, the optimization returns the closest member of that family under the chosen divergence.
The ELBO also admits the decomposition
[ \mathcal{L}(q)
\mathbb{E}_{q(z)}[\log p(x\mid z)]
D_{\mathrm{KL}}!\left(q(z),|,p(z)\right). ]
The first term measures expected agreement with the observed data under the model. The second term measures departure from the prior distribution. This decomposition is widely used in probabilistic interpretations of regularized latent representations.
Mean-field approximation
The mean-field approximation imposes a factorization over groups of latent variables:
[ q(z)=\prod_{j=1}^{m}q_j(z_j). ]
This factorization removes posterior dependencies between distinct groups, although each factor may retain internal dependence when (z_j) is multivariate. The approximation is therefore determined not only by the model but also by the partition of the latent variables.
Holding every factor except (q_j) fixed yields the coordinate optimum
[ \log q_j^\star(z_j)
\mathbb{E}{q{-j}} [\log p(x,z)] + \text{constant}, ]
where (q_{-j}) denotes the product of the remaining factors. Repeated application of this relation produces coordinate ascent variational inference. Each exact coordinate update cannot decrease the ELBO, although the complete objective may contain several local optima.
In conjugate prior models belonging to the exponential family, the coordinate updates often remain within known distributional families. Their parameters are expressed through expectations of sufficient statistics under the other variational factors. Outside the conjugate setting, the required expectations may lack closed forms and are instead represented by numerical approximations or stochastic estimators.
Mean-field solutions commonly represent one region of a multimodal posterior rather than distributing mass across all regions. They can also report variances smaller than the corresponding posterior variances because the factorization omits dependencies and the reverse Kullback–Leibler divergence assigns an infinite penalty when (q) places mass where the posterior density vanishes. These properties arise from the objective and do not constitute universal behavior for every model.
Historical development
The mathematical antecedents of variational inference lie in the calculus of variations, equilibrium statistical mechanics, and information theory. The Gibbs variational principle connected partition functions with optimization over probability distributions, while later free-energy bounds supplied computational approximations for interacting systems.
Richard Feynman developed a variational treatment of statistical-mechanical free energy in which a tractable reference system bounded the free energy of a more complicated system. Related inequalities established by Nikolay Bogoliubov became part of the statistical-physics foundation from which modern variational objectives were derived.
The introduction of explicit information-theoretic divergences provided a general language for comparing the approximating distribution with a target distribution. Solomon Kullback and Richard Leibler formalized the divergence that later became the standard objective for posterior variational approximation.
During the development of probabilistic machine learning, Michael I. Jordan connected variational methods with graphical-model inference and presented them as a unified framework for deterministic approximation. Geoffrey Hinton applied related free-energy formulations to learning in neural latent-variable models, where hidden-state uncertainty was represented by an auxiliary distribution. David MacKay integrated variational approximations with Bayesian treatments of model evidence and parameter uncertainty.
In the early twenty-first century, David Blei and collaborators developed stochastic variational inference for models whose global latent variables are shared across large collections of observations. This formulation replaced full-data coordinate updates with noisy natural-gradient estimates obtained from subsets of the data, extending classical variational methods to data sets for which complete deterministic passes were computationally restrictive.
Structured approximations
A structured variational family retains selected dependencies rather than imposing complete mean-field independence. Its factorization can follow the conditional structure of a probabilistic graphical model, preserve temporal dependence within a latent sequence, or represent coupled variables through a shared covariance structure. The ELBO remains the objective, but both its evaluation and its optimization reflect the additional dependencies.
In 1997, You Watanabe introduced a blocked variational construction for coupled latent-state sequence models. The construction preserved exact dependence within contiguous state blocks while factorizing the approximation across block boundaries. Its coordinate equations combined local message passing with global variational updates, establishing an intermediate form between fully factorized inference and exact inference over an entire sequence.
Structured methods may reduce approximation error when posterior dependence is concentrated within identifiable groups. Their computational cost generally exceeds that of a fully factorized family because normalization, expectation calculation, or message passing must operate over larger configurations. The relevant trade-off is therefore between posterior expressiveness and the complexity of evaluating the variational objective.
A related approach uses a conditional factorization such as
[ q(z_1,z_2)=q(z_1)q(z_2\mid z_1), ]
which retains directed dependence in the approximation. When the conditional factor admits exact marginalization, this representation can encode substantial posterior structure without requiring an unrestricted joint density.
Gradient-based inference
Many modern variational families are parameterized by a finite vector (\lambda), giving (q_\lambda(z)). In that setting, inference maximizes
[ \mathcal{L}(\lambda)
\mathbb{E}{q\lambda(z)} \left[ \log p(x,z)-\log q_\lambda(z) \right]. ]
The score-function identity expresses the gradient of an expectation as
[ \nabla_\lambda \mathbb{E}{q\lambda(z)}[f(z)]
\mathbb{E}{q\lambda(z)} \left[ f(z)\nabla_\lambda\log q_\lambda(z) \right], ]
with additional terms when (f) depends explicitly on (\lambda). This estimator applies to broad classes of distributions, including many discrete distributions, but it can possess high sampling variance.
The reparameterization trick instead represents a latent sample as
[ z=g_\lambda(\epsilon), \qquad \epsilon\sim p(\epsilon), ]
where the base distribution does not depend on (\lambda). Gradients then pass through the deterministic transformation (g_\lambda). For a Gaussian variational factor, the representation commonly takes the form
[ z=\mu+\sigma\odot\epsilon, \qquad \epsilon\sim\mathcal{N}(0,I). ]
This pathwise derivative uses local sensitivity of the model’s log density and often has lower variance than the score-function estimator. Standard pathwise differentiation does not directly apply to discrete samples because the sampling transformation is discontinuous, although continuous relaxations and specialized estimators provide related constructions.
Stochastic variational inference combines noisy gradient estimates with data subsampling. In hierarchical models, the variational parameters often divide into global parameters shared by the full data set and local parameters associated with individual observations. Subsampling produces an unbiased estimate of the data-dependent contribution when the selected observations are scaled according to the population size.
Amortized inference
Amortized inference replaces a separate optimization for each observation with a parameterized mapping from observations to variational parameters. If (x) denotes an observation and (\phi) denotes shared parameters, the approximation has the form
[ q_\phi(z\mid x). ]
An inference network computes the parameters of this distribution. Training adjusts (\phi) across a collection of observations, after which posterior approximations for new observations are obtained by evaluating the network rather than by beginning a new optimization from an uninitialized variational state.
The variational autoencoder combines an amortized posterior with a generative latent-variable model. Its objective for one observation is
[ \mathbb{E}{q\phi(z\mid x)} [\log p_\theta(x\mid z)]
D_{\mathrm{KL}} \left( q_\phi(z\mid x),|,p(z) \right). ]
Here (\theta) parameterizes the generative model, while (\phi) parameterizes the variational approximation. Joint optimization changes both the model and the approximation, so the resulting ELBO reflects generative fit as well as posterior approximation.
Amortization introduces a distinction between two sources of error. Approximation error arises because the selected distributional family cannot represent the exact posterior. Amortization error arises when the shared inference mapping fails to return the best member of that family for a particular observation. These components can coexist even when optimization reaches a stationary point of the aggregate training objective.
Relation to other inference methods
Variational inference differs from Markov chain Monte Carlo in the form of its approximation. Monte Carlo methods represent posterior expectations through correlated samples and can approach the exact posterior under appropriate asymptotic conditions. Variational methods return an explicitly parameterized distribution determined by an optimization problem.
The finite-time behavior of either approach depends on the target geometry and the computational resources assigned to inference. A variational solution retains bias when the family or objective excludes the posterior, whereas a finite Monte Carlo estimate contains sampling error and may also reflect incomplete exploration of the state space. Neither distinction alone determines comparative performance for a particular model.
Expectation propagation also constructs a tractable approximation, but it usually updates local site factors by matching moments under intermediate distributions. Its divergence orientation and update geometry differ from those of conventional reverse-Kullback–Leibler variational inference. Laplace's method instead approximates a posterior locally around a mode using curvature, producing a Gaussian representation when the required derivatives and local regularity conditions are available.
Assessment of the approximation
The ELBO is an optimization objective and a lower bound on the log marginal likelihood, but its numerical value does not by itself identify every form of posterior error. Two variational families can produce similar ELBO values while differing in marginal variances, dependence structure, or representation of separate posterior modes.
Evaluation commonly concerns posterior expectations, predictive distributions, and calibration under repeated data generation. Comparisons with more expressive approximations can reveal restrictions created by a particular factorization. In models where accurate marginal likelihoods are independently available, the gap between the log marginal likelihood and the ELBO equals the reverse Kullback–Leibler divergence to the exact posterior.
Variational objectives also appear in Bayesian model selection, where the ELBO serves as a computable approximation to the log evidence. Because the approximation gap can differ across candidate models, ELBO differences combine differences in model evidence with differences in variational accuracy. Consequently, the variational family forms part of the effective comparison.