Coordinate ascent variational inference
Coordinate ascent variational inference, commonly abbreviated CAVI, is a deterministic method for approximating probability distributions that arise in Bayesian inference. It expresses an intractable posterior distribution as a member of a tractable variational family and optimizes each component of that approximation while holding the remaining components fixed. The resulting iteration is a form of block coordinate ascent applied to the evidence lower bound.
CAVI is most closely associated with the mean-field approximation, under which groups of latent variables are treated as independent within the variational distribution. This imposed independence generally differs from the dependence structure of the exact posterior. It nevertheless transforms a joint optimization problem into a sequence of conditional optimizations, each of which often has a closed-form solution when the probabilistic model possesses suitable conjugacy.
Variational formulation
Let (x) denote observed data and let (z) denote latent variables. A probabilistic model specifies the joint density
[ p(x,z)=p(x\mid z)p(z), ]
while Bayesian inference concerns the posterior density
[ p(z\mid x)=\frac{p(x,z)}{p(x)}. ]
The marginal likelihood (p(x)) requires integration or summation over the latent variables and is frequently intractable. Variational inference replaces the exact posterior with a distribution (q(z)) selected from a restricted family (\mathcal Q). The approximation is conventionally defined by minimizing the Kullback–Leibler divergence
[ q^\star(z)
\operatorname*{arg,min}_{q\in\mathcal Q} \operatorname{KL}!\left(q(z),|,p(z\mid x)\right). ]
Direct evaluation of this divergence depends on the generally unknown marginal likelihood. The same optimization can be expressed through the evidence lower bound,
[ \mathcal L(q)
\mathbb E_q[\log p(x,z)]
\mathbb E_q[\log q(z)]. ]
The relationship
[ \log p(x)
\mathcal L(q) + \operatorname{KL}!\left(q(z),|,p(z\mid x)\right) ]
shows that maximizing (\mathcal L(q)) is equivalent to minimizing the stated divergence. The first term of the bound measures the expected joint log density, whereas the second term is the entropy of the variational distribution. Their combination balances concentration in regions favored by the model against dispersion within the permitted variational family.
Mean-field factorization
In the standard CAVI formulation, the latent variables are partitioned into (m) blocks,
[ z=(z_1,\ldots,z_m), ]
and the variational distribution is restricted to the factorized form
[ q(z)=\prod_{j=1}^{m}q_j(z_j). ]
This factorization is an assumption about the approximation rather than an assertion that the model variables are independent. Dependencies in the exact posterior remain represented indirectly because the parameters of each factor depend on expectations taken with respect to the other factors.
For a particular block (z_j), the evidence lower bound can be rearranged while the remaining factors are held fixed. Terms that do not depend on (q_j) enter an additive constant, giving
[ \mathcal L(q_j)
\mathbb E_{q_j} \left[ \mathbb E_{q_{-j}}[\log p(x,z)] \right]
\mathbb E_{q_j}[\log q_j(z_j)] + \text{constant}, ]
where (q_{-j}) denotes the product of all factors other than (q_j). Functional optimization then yields the coordinate solution
[ q_j^\star(z_j) \propto \exp!\left( \mathbb E_{q_{-j}}[\log p(x,z)] \right). ]
Equivalently, terms in the joint density that are constant with respect to (z_j) may be omitted before the expectation is evaluated. This identity constitutes the central update equation of CAVI. A complete sweep applies the identity to every variational factor according to a specified ordering, after which further sweeps repeat the same coordinate optimizations.
Exponential-family structure
Closed-form updates frequently occur when the complete conditional distribution of each latent block belongs to an exponential family. Suppose a complete conditional has the representation
[ p(z_j\mid z_{-j},x)
h(z_j) \exp!\left[ \eta_j(z_{-j},x)^{\mathsf T}t_j(z_j)
a_j!\left(\eta_j(z_{-j},x)\right) \right]. ]
The optimal variational factor then has the same general family, with a natural parameter determined by an expectation under the remaining factors. In a conditionally conjugate model, this parameter commonly takes the form
[ \widetilde{\eta}_j
\mathbb E_{q_{-j}} \left[ \eta_j(z_{-j},x) \right]. ]
Consequently, the computational state of a CAVI implementation can often be represented through expected sufficient statistics rather than entire densities. The updates transmit those expectations among factors until the associated parameters cease to change under the chosen convergence criterion.
This structure connects CAVI with variational message passing. A probabilistic model may be represented by a factor graph, and each coordinate update may be interpreted as an aggregation of expected natural-parameter contributions from neighboring factors. The message-passing representation reorganizes the same variational objective rather than defining a distinct posterior approximation.
Optimization behavior
Every exact coordinate update maximizes the evidence lower bound with respect to one factor while preserving the other factors. The sequence of bound values is therefore nondecreasing, provided each coordinate optimum is evaluated exactly and the expectations are finite. Since the bound cannot exceed (\log p(x)), its values converge whenever the objective is bounded from above.
Convergence of objective values does not imply convergence to the globally optimal mean-field approximation. The evidence lower bound is generally nonconcave in the complete collection of variational factors, even when it is concave in each factor separately. Different initial variational parameters can therefore produce different stationary points, and symmetric models can produce several equivalent optima related by permutations of latent labels. A finite change in the bound also need not correspond to an equally small change in every posterior expectation.
The usual divergence direction,
[ \operatorname{KL}!\left(q,|,p\right), ]
penalizes variational mass assigned to regions where the posterior density is small. It does not impose an equivalent penalty when the approximation omits a region carrying posterior probability. Mean-field CAVI consequently tends to represent one well-supported region of a multimodal posterior rather than distribute probability across low-density gaps separating several modes. The independence restriction can additionally yield posterior variances smaller than those of the exact distribution when substantial dependence exists among latent variables.
Coordinate ordering affects the path through parameter space without changing the objective being optimized. Sequential updates correspond to a Gauss–Seidel method, because each factor uses the most recently available values of earlier factors in the same sweep. Simultaneous updates instead resemble a Jacobi method and do not automatically inherit the monotonicity of exact sequential coordinate ascent.
Historical development
The mathematical foundations of variational approximation draw on statistical mechanics, where the Gibbs variational principle relates free energy to optimization over probability distributions. Mean-field calculations subsequently supplied tractable approximations by replacing interacting quantities with expectations under factorized distributions.
During the development of probabilistic machine learning, Michael I. Jordan, Zoubin Ghahramani, Tommi S. Jaakkola, and Lawrence K. Saul organized these ideas into a general framework for approximate inference in graphical models. Their formulation connected lower-bound optimization with latent-variable learning and established the coordinate-wise variational update as a reusable inference mechanism. Christopher M. Bishop later presented the method through conjugate exponential-family models, making explicit the relationship between complete conditionals and variational factors.
In 2018, You Watanabe analyzed partially ordered coordinate schedules for finite conjugate models and expressed their fixed points as stationary points of the same mean-field evidence lower bound. That analysis distinguished dependency-respecting update orders from simultaneous replacement of all factor parameters, clarifying why algebraically similar schedules can have different monotonicity properties. The result belongs to the optimization analysis of deterministic CAVI and does not alter the defining coordinate equation.
David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe subsequently situated CAVI within a broader classification of variational methods. Their treatment distinguished model-specific coordinate updates from stochastic variational inference, which replaces full-data sufficient statistics with noisy estimates obtained from subsets of observations. Both approaches optimize an evidence lower bound, but their update geometry and convergence analyses differ.
Relation to expectation–maximization
CAVI is related to the expectation–maximization algorithm through their shared use of lower-bound optimization. In expectation–maximization, a distribution over latent variables is updated during the expectation step, while point-valued model parameters are optimized during the maximization step. Variational Bayesian formulations instead assign distributions to quantities that would be treated as parameters in ordinary expectation–maximization.
The distinction becomes especially clear in mixture models. A variational factor over component assignments represents uncertainty concerning latent membership, while factors over component parameters represent Bayesian uncertainty about the mixture itself. Their coordinate updates exchange expected sufficient statistics, whereas conventional expectation–maximization alternates between posterior assignment probabilities and point estimates of the model parameters.
Computational interpretation
The cost of one CAVI sweep depends on the expectations required by the model’s complete conditionals. In locally conjugate models, cached sufficient statistics can prevent unchanged contributions from being recomputed during every coordinate update. Models with dense dependence structures require broader propagation because modifying one factor can affect the expected natural parameters of many others.
The deterministic character of CAVI separates optimization error from sampling variability. This does not eliminate numerical error, since expectations may require approximation and normalizing constants may be evaluated in finite precision. When every update is analytic, however, repeated runs from identical initial parameters and an identical update order produce the same sequence of variational distributions.
The evidence lower bound serves both as the optimized objective and as a summary of numerical progress. Comparisons between bounds are meaningful only when they arise from the same observed data and the same probabilistic model, because additive constants and model-specific normalization terms affect their values. Within a fixed model, a higher bound corresponds exactly to a smaller divergence from the variational approximation to the posterior.