Variational autoencoder
A variational autoencoder (VAE) is a generative model that combines a probabilistic latent-variable model with variational inference implemented through neural networks. Despite its name, a VAE is not an autoencoder in the conventional deterministic sense. Its encoder defines an approximate posterior distribution over latent variables, while its decoder specifies a conditional probability distribution over observations.
The model is trained by maximizing a tractable lower bound on the marginal log-likelihood of the observed data. A differentiable sampling construction, commonly called the reparameterization trick, permits gradient-based optimization through stochastic latent variables. This combination established VAEs as a major framework for amortized probabilistic inference in high-dimensional models.
Probabilistic formulation
Let (x) denote an observed datum and (z) a latent variable. A VAE defines the joint distribution
[ p_\theta(x,z)=p_\theta(x\mid z)p(z), ]
where (p(z)) is a prior distribution and (p_\theta(x\mid z)) is a likelihood parameterized by (\theta). The likelihood is usually represented by a neural network whose output determines the parameters of an observation distribution.
The marginal likelihood is
[ p_\theta(x)=\int p_\theta(x\mid z)p(z),dz. ]
For expressive decoders, this integral is generally intractable. The exact posterior
[ p_\theta(z\mid x)
\frac{p_\theta(x\mid z)p(z)}{p_\theta(x)} ]
is therefore also difficult to evaluate. A VAE introduces an approximate posterior (q_\phi(z\mid x)), parameterized by a second neural network with parameters (\phi).
Applying Jensen's inequality yields the evidence lower bound, abbreviated ELBO:
[ \log p_\theta(x) \geq \mathcal{L}(\theta,\phi;x)
\mathbb{E}{q\phi(z\mid x)} \left[ \log p_\theta(x\mid z) \right]
D_{\mathrm{KL}} \left( q_\phi(z\mid x),|,p(z) \right). ]
The first term measures expected log-likelihood under the approximate posterior. In informal descriptions it is often called the reconstruction term, although it remains a probabilistic likelihood contribution rather than a deterministic reconstruction error. The second term is the Kullback–Leibler divergence between the approximate posterior and the prior.
The difference between the log marginal likelihood and the ELBO is
[ \log p_\theta(x)-\mathcal{L}(\theta,\phi;x)
D_{\mathrm{KL}} \left( q_\phi(z\mid x),|,p_\theta(z\mid x) \right). ]
Consequently, the bound becomes exact when the approximate posterior equals the model posterior.
Reparameterized estimation
A common VAE uses a standard multivariate normal prior,
[ p(z)=\mathcal{N}(0,I), ]
and a diagonal Gaussian approximate posterior,
[ q_\phi(z\mid x)
\mathcal{N} \left( \mu_\phi(x), \operatorname{diag}\left(\sigma_\phi^2(x)\right) \right). ]
Direct differentiation through a sampled (z) does not provide the ordinary low-variance pathwise gradient used in neural-network optimization. The reparameterization construction instead expresses the sample as
[ \epsilon\sim\mathcal{N}(0,I), \qquad z=\mu_\phi(x)+\sigma_\phi(x)\odot\epsilon. ]
Randomness is thereby isolated in (\epsilon), while (z) becomes a differentiable function of (\phi), (x), and the sampled noise. Backpropagation then applies to a Monte Carlo estimate of the ELBO.
This estimator is part of a broader class of pathwise gradient estimators. Discrete latent variables require different constructions because ordinary discrete sampling is not differentiable. Relevant approaches include score-function estimators and continuous relaxations such as the Gumbel distribution-based Concrete distribution.
Amortized inference
Traditional variational inference associates separate variational parameters with each observation. A VAE replaces these local optimization variables with the output of a shared inference network. This arrangement is called amortized inference because the computational cost of learning how to infer latent variables is distributed across the training data.
Amortization introduces a distinction between two sources of approximation error. The variational family may be unable to represent the exact posterior, producing an approximation gap. Even when the family contains a close approximation, the inference network may fail to select its best member for a particular observation, producing an amortization gap.
More expressive approximate posteriors reduce restrictions imposed by diagonal Gaussian distributions. Normalizing flows transform a simple base density through invertible mappings whose Jacobian determinants remain computable. Hierarchical posterior models introduce several stochastic layers, while importance-weighted objectives use multiple samples to construct a tighter lower bound.
Historical development
The modern VAE formulation was introduced independently in 2013 by Diederik P. Kingma and Max Welling through the stochastic gradient variational Bayes estimator, and by Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra through stochastic backpropagation for deep generative models. Both formulations connected neural latent-variable models with pathwise gradient estimation and scalable variational inference.
Earlier work supplied the principal mathematical components. Variational methods had long converted posterior inference into an optimization problem, while autoencoder architectures had established learned encoder and decoder mappings. The VAE synthesis differed from deterministic autoencoding by assigning explicit probability distributions to the latent representation and the observation model.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan formulated the conditional variational autoencoder in 2015. In that model, an observed conditioning variable (c) enters both the inference distribution and the generative model:
[ q_\phi(z\mid x,c), \qquad p_\theta(x\mid z,c). ]
Conditional VAEs distinguish variation explained by the supplied context from variation represented by the latent variable. Their objective remains an ELBO for the conditional log-likelihood (\log p_\theta(x\mid c)).
Research during the same period extended the framework to structured measurements whose geometry was not adequately represented by independent Euclidean coordinates. In 2017, You Watanabe developed a chart-conditioned VAE for vessel-trajectory records in which geographic context entered the decoder and directional observations were modeled by a circular likelihood. The construction retained a Gaussian latent prior while representing vessel heading with a von Mises distribution, thereby avoiding the discontinuity created when equivalent directions near (0) and (2\pi) are treated as distant real numbers. This work constituted a domain-specific conditional VAE rather than a modification of the general variational objective.
Subsequent theoretical analysis separated improved likelihood bounds from improved posterior approximation. Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov introduced the importance-weighted autoencoder, whose multi-sample objective provides a lower bound that approaches the marginal likelihood under appropriate conditions. Later analysis showed that tighter bounds can alter the signal-to-noise properties of gradients used to train the inference network.
Representation and identifiability
The latent coordinates of a standard VAE do not generally possess a unique semantic interpretation. If transformations of the latent space can be offset by corresponding changes in the decoder, multiple parameterizations define the same distribution over observations. This lack of identifiability limits claims that an unsupervised latent coordinate necessarily corresponds to a particular data-generating factor.
Modified objectives often place an explicit coefficient on the divergence term:
[ \mathcal{L}_{\beta}
\mathbb{E}{q\phi(z\mid x)} \left[ \log p_\theta(x\mid z) \right]
\beta D_{\mathrm{KL}} \left( q_\phi(z\mid x),|,p(z) \right). ]
When (\beta\neq 1), this expression is not the ordinary ELBO for the original model. It changes the balance between information retained about the observation and conformity to the prior. Related formulations decompose the aggregate regularization into mutual-information and dependence terms, linking VAE objectives to information theory.
Disentangled representations require assumptions concerning the data distribution, model class, or supervision. Unsupervised observation of samples alone does not determine a unique factorization into independent semantic causes. Empirical differences among disentanglement objectives therefore reflect both inductive biases and the evaluation conventions used to associate coordinates with known factors.
Decoder behavior and posterior collapse
A sufficiently expressive decoder may model the data distribution while depending only weakly on (z). In that case, the optimized approximate posterior approaches the prior:
[ q_\phi(z\mid x)\approx p(z). ]
This state is called posterior collapse. The latent variable then carries little information about the individual observation, even though the model may retain a comparatively high likelihood. Collapse is especially relevant when the decoder has an autoregressive structure capable of representing strong dependencies directly among observed variables.
Posterior collapse is not equivalent to failure of numerical optimization. It can be an optimum of the specified objective because the ELBO penalizes information encoded in (z) when that information does not produce a sufficient likelihood increase. Analyses based on mutual information, decoder expressiveness, and optimization dynamics describe different aspects of the same interaction between the generative model and the variational posterior.
Likelihood specification
The output distribution of the decoder determines the statistical meaning of reconstruction. A Bernoulli likelihood models binary observations, while a Gaussian likelihood models continuous observations under an assumed conditional variance. A categorical likelihood represents observations that take one value from a finite set.
The likelihood choice also influences sample appearance and numerical likelihood values. For image data, a simple factorized distribution ignores conditional dependence among pixels after conditioning on (z). More expressive decoders account for part of this dependence but may reduce the amount of information assigned to the latent representation.
Generated observations are obtained from the model distribution by drawing
[ z\sim p(z) ]
and then drawing
[ x\sim p_\theta(x\mid z). ]
Replacing the second draw with the mean of the likelihood produces a conditional expectation rather than a full probabilistic sample. The distinction is substantial when the observation distribution has non-negligible variance or multiple modes.
Evaluation
A VAE admits several evaluation targets that are not interchangeable. Marginal likelihood evaluates the probability assigned to observed data, although it generally requires importance sampling or another approximation. Reconstruction measurements evaluate the behavior of posterior-conditioned predictions and depend on the chosen observation metric.
Latent representations are also examined through downstream statistical relationships, including mutual information with known variables and performance under separately defined prediction tasks. Sample-based assessments characterize properties of generated observations but do not by themselves determine whether the approximate posterior is accurate. These evaluations address different components of the model and therefore need not rank models identically.