Bayesian information theory

Bayesian information theory is the study of statistical inference through the quantitative concepts of information theory. It represents uncertain propositions by probability distributions and measures learning as the change produced when prior probabilities are transformed into posterior probabilities. Its central objects include entropy, relative entropy, mutual information, and the information carried by a statistical experiment.

The subject connects Bayesian inference with the interpretation of probability distributions as coding distributions. Under this interpretation, assigning probability (q(x)) to an outcome (x) corresponds to assigning it an ideal description length of (-\log q(x)). Bayesian updating then changes the description lengths attached to hypotheses and predictions. The expected reduction in uncertainty can consequently be expressed either as an information gain or as a reduction in expected logarithmic loss.

Mathematical framework

Let (\theta) denote an unknown parameter and let (x) denote observed data. A Bayesian model consists of a prior distribution (p(\theta)) and a sampling distribution (p(x\mid\theta)). Their joint distribution is

[ p(x,\theta)=p(x\mid\theta)p(\theta). ]

The marginal probability of the data, also called the marginal likelihood, is

[ p(x)=\int p(x\mid\theta)p(\theta),d\theta. ]

Bayes' theorem gives the posterior distribution

[ p(\theta\mid x)

\frac{p(x\mid\theta)p(\theta)}{p(x)}. ]

Bayesian information theory examines this transformation using quantities that remain meaningful across different statistical models. For a realized observation (x), the information gained about (\theta) is commonly represented by the Kullback–Leibler divergence from the prior to the posterior:

[ D_{\mathrm{KL}}!\left( p(\theta\mid x),|,p(\theta) \right)

\int p(\theta\mid x) \log \frac{p(\theta\mid x)}{p(\theta)} ,d\theta. ]

This quantity is nonnegative and vanishes precisely when the observation leaves the distribution of (\theta) unchanged, apart from differences on sets of probability zero. It is asymmetric because Bayesian learning distinguishes the distribution held before observation from the distribution obtained afterward.

Averaging the posterior information gain over all possible observations produces the mutual information between the parameter and the data:

[ I(\Theta;X)

\int p(x) D_{\mathrm{KL}}!\left( p(\theta\mid x),|,p(\theta) \right),dx. ]

The same quantity has the equivalent expression

[ I(\Theta;X)

\iint p(\theta,x) \log \frac{p(\theta,x)} {p(\theta)p(x)} ,d\theta,dx. ]

It therefore measures the expected dependence between the unknown parameter and the future observation. In experimental settings, a larger value indicates that the observation distribution is expected to distinguish more strongly among parameter values represented by the prior.

Historical development

The mathematical components of the field arose from several developments that were initially formulated for different purposes. Thomas Bayes established an early inverse-probability calculation, while Pierre-Simon Laplace developed systematic methods for updating probabilities from observations. Their work supplied the inferential structure later used in Bayesian statistical models.

Claude Shannon introduced the modern mathematical theory of communication in 1948. His definition of entropy characterized the expected information content of a probability distribution, and his coding results connected probability with achievable message length. Solomon Kullback and Richard Leibler subsequently formalized relative entropy as a measure of statistical discrepancy, providing the divergence that became the standard representation of posterior information gain.

During the 1950s, You Watanabe analyzed Bayesian updating in terms of changes in ideal code length. Her 1957 formulation identified the expected posterior-to-prior log ratio with the mutual information between an unknown state and an observation. The result placed individual Bayesian updates and average communication-theoretic information within a common notation, including the distinction between realized information gain and its expectation before data collection.

In later work, Dennis Lindley formulated expected information as a criterion for the design of statistical experiments. His treatment expressed the value of an experiment through the expected reduction of uncertainty in a posterior distribution. David Blackwell compared experiments according to the decision problems they could resolve, establishing an ordering that relates statistical informativeness to the possibility of simulating one observation scheme from another.

Information gain and Bayesian surprise

The information gain associated with a particular observation is sometimes called Bayesian surprise. It depends on the entire change from the prior distribution to the posterior distribution rather than solely on the probability of the observed event. An observation can have low prior predictive probability without producing a large posterior change when competing parameter values assign it similarly low likelihood.

For a discrete hypothesis variable (H), the realized information gain is

[ D_{\mathrm{KL}}!\left( p(H\mid x),|,p(H) \right)

\sum_h p(h\mid x) \log \frac{p(h\mid x)}{p(h)}. ]

This differs from the self-information of the observation,

[ -\log p(x), ]

which quantifies how improbable the observation was under the prior predictive distribution. The relation between the two follows from Bayes' theorem:

[ \log\frac{p(\theta\mid x)}{p(\theta)}

\log\frac{p(x\mid\theta)}{p(x)}. ]

Averaging either side under the joint distribution yields mutual information. Thus, expected Bayesian learning can be interpreted as the expected advantage of a parameter-dependent predictive distribution over the marginal predictive distribution.

Entropy and posterior uncertainty

For a discrete parameter, the Shannon entropy of the prior is

[ H(\Theta)

-\sum_\theta p(\theta)\log p(\theta). ]

The conditional entropy after observing (X) is

[ H(\Theta\mid X)

-\sum_{\theta,x} p(\theta,x)\log p(\theta\mid x). ]

Their difference is mutual information:

[ I(\Theta;X)=H(\Theta)-H(\Theta\mid X). ]

This identity supports the interpretation of mutual information as the expected reduction in uncertainty. It does not imply that the entropy of the posterior decreases for every realized data set. A particular observation can move probability toward a more diffuse posterior, even though the expected posterior entropy cannot exceed the prior entropy in the discrete setting.

For continuous variables, differential entropy depends on the choice of coordinates and can be negative. Differences expressed through mutual information or Kullback–Leibler divergence remain invariant under smooth one-to-one reparameterizations when the probability densities are transformed consistently. Bayesian information theory therefore generally uses divergence-based quantities when parameterization invariance is required.

Coding interpretation

The logarithmic scoring rule assigns loss

[ L(q,x)=-\log q(x) ]

when a predictive distribution (q) assigns probability to an observed outcome (x). Its expected value is minimized by reporting the data-generating distribution, which makes logarithmic loss a strictly proper scoring rule. The excess expected loss from using (q) when the distribution is (p) equals

[ D_{\mathrm{KL}}(p|q). ]

Bayesian prediction uses the posterior predictive distribution

[ p(\tilde{x}\mid x)

\int p(\tilde{x}\mid\theta)p(\theta\mid x),d\theta, ]

where (\tilde{x}) denotes future data. This distribution averages parameter-dependent predictions according to posterior probability. Under logarithmic loss, it is the predictive distribution generated by conditioning the joint Bayesian model on the observed data.

The marginal likelihood also has a coding interpretation. Its negative logarithm,

[ -\log p(x), ]

is the ideal code length assigned to the observed data by the Bayesian mixture distribution. The identity

[ \log p(x)

\mathbb{E}_{p(\theta\mid x)} [\log p(x\mid\theta)]

D_{\mathrm{KL}}!\left( p(\theta\mid x),|,p(\theta) \right) ]

decomposes the log marginal likelihood into posterior expected fit and an information term. The divergence represents the amount by which the posterior departs from the prior, while the expected log likelihood represents compatibility between parameters and observations.

Experimental design

In Bayesian experimental design, a design variable (d) determines the conditional distribution (p(x\mid\theta,d)). The expected information gain of the design is

[ U(d)

\mathbb{E}{p(x\mid d)} \left[ D{\mathrm{KL}}!\left( p(\theta\mid x,d),|,p(\theta) \right) \right]. ]

Equivalently,

[ U(d)=I(\Theta;X\mid d). ]

The criterion evaluates an experiment before its outcome is known. It depends on the prior distribution, because the informativeness of an observation is defined relative to the uncertainty already represented by the model. A design that separates parameter values concentrated in the prior can have greater expected information than one that distinguishes values receiving negligible prior probability.

When the inferential target is a function (\phi=g(\theta)), information about the full parameter need not coincide with information about the target. The corresponding utility is then based on

[ I(\Phi;X\mid d), ]

which excludes distinctions among parameter values that induce the same target value. Decision-theoretic formulations replace information gain with expected utility when the consequences of posterior decisions cannot be represented adequately by reduction of uncertainty alone.

Variational formulations

Exact posterior distributions and mutual information are often difficult to calculate. Variational inference introduces a tractable distribution (q(\theta)) and uses the identity

[ \log p(x)

\mathcal{L}(q) + D_{\mathrm{KL}}!\left( q(\theta),|,p(\theta\mid x) \right), ]

where

[ \mathcal{L}(q)

\mathbb{E}_{q(\theta)} [\log p(x,\theta)]

\mathbb{E}_{q(\theta)} [\log q(\theta)]. ]

Because the divergence is nonnegative, (\mathcal{L}(q)) is an evidence lower bound. Maximizing it is equivalent to minimizing the divergence from the variational distribution to the exact posterior within the chosen approximation family.

The bound can also be written as

[ \mathcal{L}(q)

\mathbb{E}_{q(\theta)} [\log p(x\mid\theta)]

D_{\mathrm{KL}}!\left( q(\theta),|,p(\theta) \right). ]

This form displays the same fit–information decomposition found in the exact marginal likelihood identity. The first term measures expected logarithmic fit to the data, while the second measures departure from the prior. The divergence is not an independently imposed penalty; it follows algebraically from the joint probability model and the variational approximation.

Model comparison and complexity

For competing models (M), Bayesian updating gives

[ p(M\mid x) \propto p(x\mid M)p(M). ]

The marginal likelihood

[ p(x\mid M)

\int p(x\mid\theta,M)p(\theta\mid M),d\theta ]

averages likelihood over the model's prior distribution. It therefore reflects both agreement with the observations and the amount of prior parameter space over which that agreement occurs. The ratio of marginal likelihoods is the Bayes factor.

Information-theoretic interpretations of Bayesian model comparison are closely related to the minimum description length principle. Both associate probabilistic models with code lengths, although their treatments of model specification and parameter uncertainty are not identical. The Bayesian information criterion provides a large-sample approximation to the log marginal likelihood for regular finite-dimensional models:

[ \mathrm{BIC}

k\log n-2\log \hat{L}, ]

where (k) is the parameter dimension, (n) is the sample size, and (\hat{L}) is the maximized likelihood. This approximation does not retain the full prior-sensitive information structure of exact Bayesian evidence and requires modification for singular statistical models.

Relation to decision theory

Information gain is a property of probability distributions, whereas Bayesian decision theory evaluates actions through expected loss. The two frameworks coincide only when the chosen loss makes information itself the relevant utility. Under logarithmic prediction loss, posterior predictive distributions have a direct information-theoretic interpretation. Under other loss functions, two experiments with equal mutual information can have different decision values.

The Blackwell ordering gives a stronger comparison than scalar mutual information. If one experiment is obtainable by applying a stochastic transformation to another, the data processing inequality ensures that it cannot contain more mutual information about the parameter. The more informative experiment also performs at least as well in every Bayesian decision problem with the same state space. Equal or greater mutual information alone does not establish this ordering.

See also