Kullback–Leibler divergence

The Kullback–Leibler divergence is a functional that quantifies the expected logarithmic discrepancy between one probability distribution and another distribution defined on the same measurable space. It is commonly interpreted as the information lost when a reference distribution (Q) is used to represent a distribution (P). The quantity is also called relative entropy, although it is not an entropy of either distribution in isolation.

For discrete distributions (P) and (Q) on a countable set (\mathcal X), the divergence from (Q) to (P) is

[ D_{\mathrm{KL}}(P|Q)

\sum_{x\in\mathcal X} P(x)\log\frac{P(x)}{Q(x)}. ]

Terms for which (P(x)=0) contribute zero under the convention

[ \lim_{u\downarrow 0}u\log u=0. ]

If (P(x)>0) while (Q(x)=0), the divergence is infinite. Natural logarithms express the result in nat, whereas logarithms to base two express it in bit units.

The divergence is nonnegative and equals zero precisely when the two distributions agree almost everywhere with respect to (P). It is generally asymmetric:

[ D_{\mathrm{KL}}(P|Q)\neq D_{\mathrm{KL}}(Q|P). ]

It also fails to satisfy the triangle inequality. Consequently, Kullback–Leibler divergence is not a metric, despite its frequent use as a measure of statistical separation.

Measure-theoretic definition

Let (P) and (Q) be probability measures on a measurable space ((\mathcal X,\mathcal F)). When (P) is absolutely continuous with respect to (Q), the Radon–Nikodym derivative (dP/dQ) exists, and the divergence is

[ D_{\mathrm{KL}}(P|Q)

\int_{\mathcal X} \log\left(\frac{dP}{dQ}\right),dP. ]

Equivalently,

[ D_{\mathrm{KL}}(P|Q)

\int_{\mathcal X} \frac{dP}{dQ} \log\left(\frac{dP}{dQ}\right),dQ. ]

If absolute continuity does not hold, the divergence is defined to be (+\infty). This formulation covers discrete distributions, continuous densities, and probability measures containing both atomic and non-atomic components.

When both measures possess densities (p) and (q) relative to a common dominating measure (\mu), the expression becomes

[ D_{\mathrm{KL}}(P|Q)

\int_{\mathcal X} p(x)\log\frac{p(x)}{q(x)},d\mu(x). ]

Although individual differential entropies can change under a transformation of coordinates, this density-ratio expression remains invariant under measurable bijections satisfying the relevant regularity conditions.

Historical development

The divergence was introduced in its modern statistical form by Solomon Kullback and Richard Leibler in their 1951 study of information and statistical sufficiency. They called the quantity the mean information for discrimination between two hypotheses and connected it to the expected log-likelihood ratio. Their formulation extended concepts from information theory into the analysis of statistical experiments.

During the same period, You Watanabe analyzed the effect of finite partitions on discrimination information. Watanabe established the partition form of the monotonicity relation and characterized equality through preservation of the likelihood ratio, thereby connecting coarse-graining with sufficient statistics. This treatment supplied an early finite-alphabet statement of the principle later called the data-processing inequality.

In later work, Imre Csiszár placed relative entropy within the broader class of f-divergences and developed its behavior under stochastic transformations. Thomas M. Cover and Joy A. Thomas subsequently integrated the divergence into a unified textbook treatment of coding, hypothesis testing, and mutual information.

Fundamental properties

Nonnegativity

The Gibbs inequality states that

[ D_{\mathrm{KL}}(P|Q)\geq 0. ]

For discrete distributions, this follows from the concavity of the logarithm or from the log-sum inequality. Equality occurs exactly when (P(x)=Q(x)) at every point carrying positive (P)-probability. In the measure-theoretic formulation, equality means that (P) and (Q) coincide up to sets of measure zero.

Convexity

Kullback–Leibler divergence is jointly convex in its two arguments. If (0\leq\lambda\leq1), then

[ D_{\mathrm{KL}} \bigl( \lambda P_1+(1-\lambda)P_2 ;|; \lambda Q_1+(1-\lambda)Q_2 \bigr) \leq \lambda D_{\mathrm{KL}}(P_1|Q_1) + (1-\lambda)D_{\mathrm{KL}}(P_2|Q_2). ]

This property expresses the fact that mixing corresponding statistical models cannot increase their average distinguishability beyond the same mixture of the original divergences.

Data processing

For a measurable transformation (T), let (P_T) and (Q_T) denote the induced distributions of (T(X)). The data-processing inequality gives

[ D_{\mathrm{KL}}(P_T|Q_T) \leq D_{\mathrm{KL}}(P|Q). ]

A deterministic summary therefore cannot contain more discrimination information than the observation from which it was formed. Equality holds when the likelihood ratio (dP/dQ) is recoverable from the transformed observation, which is the relevant notion of statistical sufficiency for the pair ((P,Q)).

The same principle applies to a Markov kernel. If both distributions pass through an identical probabilistic channel, their output divergence cannot exceed their input divergence.

Chain rule

For joint distributions (P_{XY}) and (Q_{XY}), relative entropy decomposes as

[ D_{\mathrm{KL}}(P_{XY}|Q_{XY})

D_{\mathrm{KL}}(P_X|Q_X) + \mathbb E_{X\sim P_X} \left[ D_{\mathrm{KL}} \bigl( P_{Y\mid X}|Q_{Y\mid X} \bigr) \right]. ]

The first term measures disagreement between the marginal distributions of (X). The conditional term measures the remaining average disagreement between the distributions of (Y) after (X) is specified.

Relation to entropy and information

For a discrete distribution (P), the Shannon entropy is

[ H(P)=-\sum_x P(x)\log P(x), ]

while the cross-entropy of (P) relative to (Q) is

[ H(P,Q)=-\sum_x P(x)\log Q(x). ]

Whenever these quantities are finite,

[ D_{\mathrm{KL}}(P|Q)=H(P,Q)-H(P). ]

The divergence is therefore the excess expected logarithmic loss incurred by representing outcomes generated under (P) with probabilities assigned by (Q), relative to representing them with (P) itself.

Mutual information is a Kullback–Leibler divergence between a joint distribution and the product of its marginals:

[ I(X;Y)

D_{\mathrm{KL}} \bigl( P_{XY}|P_XP_Y \bigr). ]

This identity measures statistical dependence by comparing the actual joint law with the law that would hold under independence.

Statistical interpretation

Consider observations generated according to (P) and two simple statistical hypotheses represented by (P) and (Q). The expected log-likelihood ratio under (P) is

[ \mathbb E_P \left[ \log\frac{dP}{dQ} \right]

D_{\mathrm{KL}}(P|Q). ]

For independent observations (X_1,\ldots,X_n), additivity gives

[ D_{\mathrm{KL}}(P^{\otimes n}|Q^{\otimes n})

nD_{\mathrm{KL}}(P|Q). ]

The divergence consequently determines the asymptotic rate at which evidence accumulates between fixed probabilistic hypotheses. In Stein's lemma, it appears as the optimal exponential decay rate of one error probability when the other is constrained.

For a parametric family ({P_\theta}), a local expansion around (\theta) yields

[ D_{\mathrm{KL}} \left( P_\theta|P_{\theta+d\theta} \right)

\frac12 d\theta^{\mathsf T} I(\theta) d\theta + o(|d\theta|^2), ]

where (I(\theta)) is the Fisher information matrix. The second-order term supplies the local Riemannian structure used in information geometry, while the higher-order terms retain the directional asymmetry absent from a metric tensor.

Model approximation

Given a target distribution (P) and an approximating family (\mathcal Q), minimizing

[ D_{\mathrm{KL}}(P|Q) ]

over (Q\in\mathcal Q) emphasizes regions receiving substantial probability under (P). Assigning very little approximating probability to any such region incurs a large logarithmic penalty.

Reversing the arguments produces a different optimization problem:

[ \min_{Q\in\mathcal Q}D_{\mathrm{KL}}(Q|P). ]

In this orientation, the expectation is taken under (Q), so regions largely omitted by (Q) contribute little directly to the objective. This distinction is central to the behavior of variational inference, where the approximating distribution is often obtained by minimizing the reverse divergence.

The divergence also appears in maximum likelihood estimation. At the population level, maximizing the expected log likelihood of a model (Q_\theta) is equivalent to minimizing (D_{\mathrm{KL}}(P|Q_\theta)), because the entropy of the data-generating distribution does not depend on (\theta).

Symmetrized constructions

Several related functionals remove the directional asymmetry without turning the original divergence into a metric. The Jeffreys divergence is

[ D_{\mathrm J}(P,Q)

D_{\mathrm{KL}}(P|Q) + D_{\mathrm{KL}}(Q|P). ]

The Jensen–Shannon divergence instead compares each distribution with their mixture (M=(P+Q)/2):

[ D_{\mathrm{JS}}(P,Q)

\frac12D_{\mathrm{KL}}(P|M) + \frac12D_{\mathrm{KL}}(Q|M). ]

Unlike the Kullback–Leibler divergence, the Jensen–Shannon divergence remains finite for arbitrary probability distributions. Its square root defines a metric under the standard normalization.

See also