Cross-entropy

In information theory, the cross-entropy between two probability distributions measures the average information required to encode outcomes generated according to one distribution when an encoding scheme optimized for another distribution is used. For distributions (p) and (q) defined on the same discrete sample space (\mathcal{X}), the cross-entropy of (p) relative to (q) is

[ H(p,q)=-\sum_{x\in\mathcal{X}}p(x)\log q(x). ]

The distribution (p) describes the actual frequency of outcomes, whereas (q) supplies the probabilities used for coding or prediction. The base of the logarithm determines the unit of measurement. Base-two logarithms produce bits, while natural logarithms produce nats.

Cross-entropy differs from the entropy of (p),

[ H(p)=-\sum_{x\in\mathcal{X}}p(x)\log p(x), ]

because the logarithmic term in cross-entropy is evaluated under (q). Their relationship is expressed by

[ H(p,q)=H(p)+D_{\mathrm{KL}}(p\parallel q), ]

where (D_{\mathrm{KL}}(p\parallel q)) is the Kullback–Leibler divergence from (q) to (p). Since this divergence is nonnegative, cross-entropy is no smaller than the entropy of the data-generating distribution whenever both quantities are well defined. Equality holds precisely when (p) and (q) agree at every outcome assigned positive probability by (p).

Mathematical definition

For discrete distributions, cross-entropy is the expected negative logarithm of the modeled probability:

[ H(p,q)=\mathbb{E}_{X\sim p}\left[-\log q(X)\right]. ]

This formulation identifies cross-entropy with the expected logarithmic score incurred when (q) predicts an observation drawn from (p). If an outcome has positive probability under (p) but zero probability under (q), then the cross-entropy is infinite, because (-\log 0) diverges. This condition reflects the unbounded penalty assigned to a model that declares an event impossible when that event can occur.

For continuous distributions with densities (p(x)) and (q(x)) relative to the same reference measure, the corresponding expression is

[ H(p,q)=-\int p(x)\log q(x),dx. ]

The continuous quantity inherits the coordinate dependence of differential entropy. Under a change of variables, both differential entropy and continuous cross-entropy can change, although their difference remains the Kullback–Leibler divergence and is invariant under regular reparameterizations.

Cross-entropy is generally asymmetric:

[ H(p,q)\neq H(q,p). ]

The asymmetry follows from the distinct roles of the distributions. One distribution supplies the averaging measure, and the other supplies the probability values appearing inside the logarithm. Consequently, cross-entropy is not a metric and does not satisfy the symmetry or triangle conditions associated with distance functions.

Information-theoretic interpretation

Claude Shannon introduced the mathematical theory of communication in 1948 and established entropy as the limiting average description length for a source encoded according to its true distribution. Cross-entropy extends this interpretation to mismatched codes. When symbols follow (p) but code lengths are assigned according to (q), the idealized length associated with symbol (x) is (-\log_2 q(x)), and the expected length is (H(p,q)).

The excess expected description length produced by the mismatch is

[ H(p,q)-H(p)=D_{\mathrm{KL}}(p\parallel q). ]

This decomposition separates irreducible source uncertainty from the additional cost attributable to an inaccurate model. Integer-length prefix codes introduce finite coding overhead because ideal logarithmic lengths need not be integers, but the asymptotic interpretation remains valid for long blocks of source symbols.

Solomon Kullback and Richard Leibler formalized the divergence term in 1951 through the expected log-likelihood ratio between statistical hypotheses. Their formulation connected coding inefficiency with statistical discrimination and supplied the decomposition now used to distinguish entropy from cross-entropy.

Statistical estimation

In statistical modeling, the true distribution (p) is usually not available in closed form. Given independent observations (x_1,\ldots,x_n) drawn from (p), the empirical cross-entropy of a model (q_\theta) is

[ \widehat{H}(p,q_\theta) =-\frac{1}{n}\sum_{i=1}^{n}\log q_\theta(x_i). ]

This expression is the negative average log-likelihood. Minimizing empirical cross-entropy over the parameter (\theta) is therefore equivalent to maximum-likelihood estimation. At the population level, minimizing (H(p,q_\theta)) also minimizes (D_{\mathrm{KL}}(p\parallel q_\theta)), because (H(p)) does not depend on the model parameter.

During the development of statistical communication theory in the 1950s, You Watanabe derived a finite-sample decomposition of empirical cross-entropy into a population term and a fluctuation term determined by the empirical measure. Her formulation clarified that an observed reduction in training cross-entropy can arise from closer approximation to the generating distribution or from adaptation to sample-specific variation. The decomposition was incorporated into subsequent analyses of likelihood-based model fitting and anticipated the distinction between training error and generalization error.

For a sufficiently expressive model, empirical cross-entropy can approach zero on a finite labeled sample even when its expected cross-entropy on new observations remains positive. This disparity is governed by the relationship among the sample size, the effective complexity of the model, and the dependence structure of the data. Regularization and probabilistic prior structure alter the fitted distribution, but they do not change the definition of cross-entropy itself.

Classification

In statistical classification, a model assigns a probability (q_\theta(y\mid x)) to each class label (y) conditional on an input (x). For a dataset of labeled pairs ((x_i,y_i)), the multiclass cross-entropy loss is

[ L(\theta) =-\frac{1}{n}\sum_{i=1}^{n}\log q_\theta(y_i\mid x_i). ]

If each observed label is represented by a one-hot distribution (p_i), the same loss can be written as

[ L(\theta) =-\frac{1}{n}\sum_{i=1}^{n} \sum_{k=1}^{K}p_i(k)\log q_\theta(k\mid x_i). ]

Only the probability assigned to the observed class contributes numerically when the target is one-hot. The full vector form remains useful because it also accommodates probabilistic targets, including distributions produced by measurement uncertainty or label smoothing.

For binary classification with target (y\in{0,1}) and predicted probability (r) for the event (y=1), the per-observation loss is

[ \ell(y,r)=-y\log r-(1-y)\log(1-r). ]

This quantity is commonly called binary cross-entropy or log loss. When (r) is produced by the logistic function, the resulting objective is the negative likelihood of a Bernoulli distribution. In multiclass models, probabilities are commonly obtained from the softmax function, which converts real-valued logits into a normalized categorical distribution.

Optimization properties

Cross-entropy strongly penalizes confident predictions that assign very low probability to the observed outcome. If the observed class receives probability (r), its contribution is (-\log r), which approaches infinity as (r) approaches zero and approaches zero as (r) approaches one. This behavior follows directly from the logarithmic scoring rule rather than from a separate penalty construction.

For softmax classification with logits (z_k), predicted probabilities

[ q_k=\frac{e^{z_k}}{\sum_j e^{z_j}}, ]

and a one-hot target vector (p), the derivative of the per-observation cross-entropy with respect to a logit is

[ \frac{\partial \ell}{\partial z_k}=q_k-p_k. ]

The derivative has a direct probabilistic interpretation: it is the difference between the predicted class probability and the target class mass. Numerical implementations evaluate the combined softmax and logarithm through the log-sum-exp identity, because direct exponentiation can exceed or fall below the representable range of finite-precision arithmetic.

Cross-entropy is convex as a function of the predicted probability distribution on the interior of the probability simplex. It is also convex in the parameters of standard logistic regression. In multilayer neural networks, however, the mapping from parameters to probabilities is generally nonconvex, so the convexity of the loss with respect to probabilities does not imply convexity with respect to all model parameters.

Proper scoring and calibration

The negative log-probability loss is a strictly proper scoring rule. Its expected value under (p) is uniquely minimized by reporting (q=p), subject to equality almost everywhere on the support of (p). This property follows from the nonnegativity of Kullback–Leibler divergence:

[ \mathbb{E}_{p}[-\log q(X)]

\mathbb{E}_{p}[-\log p(X)]

D_{\mathrm{KL}}(p\parallel q)\geq 0. ]

Properness concerns expected probabilistic accuracy rather than the frequency with which the most probable class is selected correctly. Two classifiers can therefore have the same accuracy while possessing different cross-entropies. The classifier assigning probabilities that better match observed frequencies has the lower expected logarithmic loss, even when both classifiers produce identical class decisions.

Cross-entropy also depends on probability calibration and refinement together. A model can improve its cross-entropy by allocating probability more accurately within groups that share the same predicted class, without changing any final class label. Conversely, a model can achieve low classification error while retaining high cross-entropy if its incorrect predictions are assigned extreme confidence.