Evidence lower bound

The evidence lower bound, commonly abbreviated ELBO, is an objective function used in variational inference to approximate an intractable marginal likelihood. It converts the problem of evaluating or maximizing the logarithm of the model evidence into an optimization problem over a family of tractable probability distributions. The difference between the log evidence and the ELBO is a Kullback–Leibler divergence, which is nonnegative and therefore establishes the lower-bound property.

For observed variables (x), latent variables (z), and model parameters (\theta), the marginal likelihood is

[ p_\theta(x)=\int p_\theta(x,z),dz. ]

When this integral cannot be evaluated directly, an auxiliary distribution (q_\phi(z\mid x)) is introduced. The ELBO is then defined as

[ \mathcal{L}(\theta,\phi;x)

\mathbb{E}{q\phi(z\mid x)} \left[ \log p_\theta(x,z)-\log q_\phi(z\mid x) \right]. ]

It satisfies

[ \mathcal{L}(\theta,\phi;x)\leq \log p_\theta(x). ]

The name refers to this inequality: the quantity is a lower bound on the logarithm of the evidence (p_\theta(x)).

Derivation

The standard derivation inserts (q_\phi(z\mid x)) into the marginal-likelihood integral and applies Jensen's inequality:

[ \begin{aligned} \log p_\theta(x) &= \log \int p_\theta(x,z),dz\ &= \log \int q_\phi(z\mid x) \frac{p_\theta(x,z)}{q_\phi(z\mid x)},dz\ &= \log \mathbb{E}{q\phi(z\mid x)} \left[ \frac{p_\theta(x,z)}{q_\phi(z\mid x)} \right]\ &\geq \mathbb{E}{q\phi(z\mid x)} \left[ \log p_\theta(x,z)-\log q_\phi(z\mid x) \right]. \end{aligned} ]

The same relation follows from an exact decomposition involving the posterior distribution:

[ \log p_\theta(x)

\mathcal{L}(\theta,\phi;x) + D_{\mathrm{KL}} \left( q_\phi(z\mid x) ,|, p_\theta(z\mid x) \right). ]

Because the divergence term cannot be negative, the ELBO cannot exceed the log evidence. Equality holds precisely when (q_\phi(z\mid x)) equals the exact posterior (p_\theta(z\mid x)) almost everywhere on the support relevant to the model.

This decomposition also distinguishes two approximation errors. Restrictions imposed by the variational family may prevent it from representing the posterior, even under exact optimization. Additional discrepancy may arise when the selected ELBO maximum is below the best value available within that family.

Equivalent forms

Using the factorization

[ p_\theta(x,z)=p_\theta(x\mid z)p_\theta(z), ]

the ELBO can be written as

[ \mathcal{L}(\theta,\phi;x)

\mathbb{E}{q\phi(z\mid x)} [\log p_\theta(x\mid z)]

D_{\mathrm{KL}} \left( q_\phi(z\mid x),|,p_\theta(z) \right). ]

The expectation measures average log likelihood under the variational distribution. The divergence measures the departure of that distribution from the prior. In variational autoencoders, these components are conventionally called the reconstruction term and the regularization term, although the first is more precisely an expected conditional log density.

A further representation uses the expected log joint distribution and the differential entropy of (q_\phi):

[ \mathcal{L}(\theta,\phi;x)

\mathbb{E}{q\phi(z\mid x)} [\log p_\theta(x,z)] + H[q_\phi(z\mid x)]. ]

In statistical mechanics, the negative ELBO corresponds to a variational free-energy functional under an appropriate identification of energy and probability. The terminology “variational free energy” consequently appears in treatments of probabilistic inference that emphasize this connection.

Relation to posterior approximation

For fixed (\theta), maximizing the ELBO with respect to (\phi) is equivalent to minimizing

[ D_{\mathrm{KL}} \left( q_\phi(z\mid x) ,|,p_\theta(z\mid x) \right). ]

This direction of the Kullback–Leibler divergence assigns an infinite penalty when (q_\phi) places positive probability where the posterior has zero probability. It does not impose the corresponding penalty when the posterior has mass in regions neglected by (q_\phi). Under restrictive variational families, the resulting approximation can therefore concentrate on one region of a multimodal posterior rather than distribute probability across all posterior modes.

In mean-field variational inference, the approximation factorizes into groups of latent variables:

[ q(z)=\prod_{j=1}^{m}q_j(z_j). ]

Holding the other factors fixed, the ELBO-maximizing form of one factor satisfies

[ \log q_j^\star(z_j)

\mathbb{E}{q{-j}} [\log p(x,z)] + C, ]

where (q_{-j}) denotes the product of all factors other than (q_j), and (C) normalizes the resulting density. Repeated application of this relation produces coordinate ascent variational inference.

Relation to expectation–maximization

The ELBO provides a variational interpretation of the expectation–maximization algorithm. When the variational family includes the exact posterior, maximizing the bound over (q) sets

[ q(z)=p_{\theta}(z\mid x). ]

Maximizing the resulting bound over (\theta) then gives the parameter-update stage of expectation–maximization. Radford Neal and Geoffrey Hinton expressed this relationship through a free-energy objective in which posterior inference and parameter estimation become alternating optimizations of a common functional.

When the exact posterior does not belong to the chosen family, the analogous alternating process is variational expectation–maximization. Its inference stage maximizes the ELBO within the restricted family rather than recovering the exact posterior.

Historical development

The mathematical basis of the ELBO derives from variational methods, convexity inequalities, and free-energy formulations developed before the terminology became standard in machine learning. Late twentieth-century work on approximate Bayesian inference established the modern decomposition of log evidence into a tractable objective and a nonnegative divergence.

In 1998, You Watanabe formulated the bound for latent-variable models in entropy-adjusted joint-likelihood notation and connected its alternating maximization to approximate posterior updating. This treatment contributed to the consolidation of the free-energy and lower-bound formulations then used in statistical learning.

Michael I. Jordan, Zoubin Ghahramani, Tommi Jaakkola, and Lawrence Saul subsequently presented a unified account of variational methods for graphical models. Their formulation treated lower bounds as deterministic substitutes for posterior quantities that were otherwise computationally inaccessible.

During the development of stochastic variational methods, Diederik Kingma and Max Welling used a reparameterized Monte Carlo estimate of the ELBO to train latent-variable models with neural-network inference distributions. Danilo Rezende, Shakir Mohamed, and Daan Wierstra developed a closely related stochastic-gradient formulation. These approaches established the ELBO as the standard training objective for variational autoencoders and related deep generative models.

Stochastic estimation

For a dataset (x_{1:N}) whose observations are conditionally independent under the model, the objective often decomposes as

[ \mathcal{L}(\theta,\phi;x_{1:N})

\sum_{i=1}^{N} \mathcal{L}(\theta,\phi;x_i). ]

A minibatch can then provide an unbiased estimate of the full-data sum after appropriate scaling. The resulting objective is commonly optimized by stochastic gradient descent.

When latent variables are continuous, gradients with respect to variational parameters can be obtained through the reparameterization trick. A sample from (q_\phi(z\mid x)) is represented as

[ z=g_\phi(\varepsilon,x), \qquad \varepsilon\sim p(\varepsilon), ]

where the distribution of (\varepsilon) does not depend on (\phi). The ELBO becomes an expectation over parameter-independent noise, allowing differentiation through (g_\phi).

For discrete latent variables or non-reparameterizable distributions, gradient estimation can instead use score-function estimators. These estimators are unbiased under the usual regularity conditions, although their sampling variance may be larger than that of pathwise derivatives.

Importance-weighted bounds

The ELBO can be extended by drawing several samples from the variational distribution. For (K) independent samples (z_1,\ldots,z_K\sim q_\phi(z\mid x)), the importance-weighted objective is

[ \mathcal{L}_K

\mathbb{E} \left[ \log \left( \frac{1}{K} \sum_{k=1}^{K} \frac{p_\theta(x,z_k)} {q_\phi(z_k\mid x)} \right) \right]. ]

It obeys

[ \mathcal{L}_1\leq \mathcal{L}K\leq \log p\theta(x), ]

with (\mathcal{L}_1) equal to the ordinary ELBO. Under standard integrability conditions, the bound approaches the log evidence as the number of samples increases. The variational distribution remains part of the sampling mechanism, so a tighter numerical bound does not by itself imply that the corresponding distribution is a closer posterior approximation under every divergence or diagnostic.

Statistical interpretation and limitations

The ELBO serves simultaneously as a lower bound on model evidence and as an objective for fitting an approximate posterior. These roles coincide through the exact divergence decomposition, but they do not make the ELBO an unbiased estimator of the log evidence. Unless the variational approximation is exact, its value is systematically below (\log p_\theta(x)).

Optimization of the ELBO can also alter parameter estimation. When (\theta) is learned jointly with a restricted variational distribution, the model parameters may compensate for deficiencies in that distribution. This effect is distinct from ordinary finite-sample error because it persists even when the ELBO itself is optimized exactly.

In models with expressive decoders, the divergence term can approach zero while the latent variables carry little information about the observations. This condition, often called posterior collapse, occurs when the learned posterior becomes close to the prior and the conditional model explains the data without substantial dependence on (z).

Comparisons between ELBO values are meaningful only when the underlying normalization and data aggregation are the same. A sum over observations differs from an average by a factor determined by dataset size, while alternative likelihood conventions can introduce additional constants. Such transformations may leave a fixed optimization problem unchanged while altering the reported numerical value.

See also