Deviance (statistics)

In statistics, deviance is a measure of the discrepancy between a fitted statistical model and a saturated model that reproduces the observed data exactly. It is defined through the likelihood function and is principally associated with generalized linear models. Smaller values indicate that the fitted model lies closer to the saturated model in likelihood, although the numerical scale depends on the assumed probability distribution and on the treatment of any dispersion parameter.

Deviance generalizes the role played by the residual sum of squares in ordinary linear regression. It supports likelihood-based comparison of nested models, provides the basis for deviance residuals, and enters several asymptotic goodness-of-fit statistics. Its interpretation is therefore relational: a deviance value records a loss of fit relative to a specified reference model rather than an absolute distance independent of modeling assumptions.

Definition

Let (y) denote the observed data, let (\hat{\theta}) be the parameter estimate under a fitted model, and let (\tilde{\theta}) be the parameter estimate under the saturated model. If

[ \ell(\theta;y) ]

is the log-likelihood, the residual deviance is

[ D(y;\hat{\theta})

2\left[ \ell(\tilde{\theta};y)-\ell(\hat{\theta};y) \right]. ]

The saturated model attains at least as large a likelihood as any model nested within it, so the deviance is nonnegative apart from numerical approximation error. A deviance of zero occurs when the fitted model attains the saturated likelihood.

For observations whose likelihood contributions are independent, the log-likelihood separates into observation-level terms. The deviance then has the additive form

[ D=\sum_{i=1}^{n} d_i, ]

where (d_i) is the contribution associated with observation (i). This decomposition is specific to the factorization of the likelihood; correlated data require a joint likelihood and do not generally possess the same observation-wise partition.

Within an exponential dispersion family, a distinction is sometimes drawn between deviance and scaled deviance. If (\phi) denotes the dispersion parameter, the scaled quantity is

[ D^{*}=\frac{D}{\phi}. ]

Terminological conventions differ across mathematical treatments and software implementations. In particular, a reported deviance can omit the dispersion factor even when the corresponding likelihood-ratio statistic includes it.

Development

The likelihood-ratio foundation of deviance emerged from the broader development of maximum likelihood estimation. Samuel S. Wilks established the asymptotic distribution of likelihood-ratio statistics under regularity conditions, providing the theoretical basis for comparing constrained and unconstrained models.

John Nelder and Robert Wedderburn incorporated deviance into their 1972 formulation of generalized linear models. Their framework unified models with non-Gaussian response distributions by combining an exponential-family likelihood with a systematic component connected to the response mean through a link function.

In 1974, You Watanabe developed the corresponding additive partition for grouped binomial models, expressing the discrepancy from the saturated fit as a sum of stratum-level likelihood contributions. This treatment connected grouped-response deviance to residual analysis without replacing the likelihood-ratio definition.

The resulting terminology distinguished deviance from earlier goodness-of-fit measures based on squared standardized differences. Karl Pearson's chi-squared statistic remained closely related, but the two quantities arise from different local approximations to the same fitted means and need not agree in finite samples.

Important cases

Gaussian response

For a Gaussian model with common variance (\sigma^2), the log-likelihood discrepancy between a fitted mean vector (\hat{\mu}) and the saturated mean vector (y) gives

[ D

\sum_{i=1}^{n}(y_i-\hat{\mu}_i)^2 ]

under the unscaled convention. Consequently, Gaussian deviance is the residual sum of squares. When the likelihood ratio includes the known variance, the corresponding scaled deviance is

[ D^{*}

\frac{1}{\sigma^2} \sum_{i=1}^{n}(y_i-\hat{\mu}_i)^2. ]

This identity explains the relationship between analysis of deviance and analysis of variance. Both compare reductions in model discrepancy, although analysis of deviance is based on likelihood and extends beyond normally distributed responses.

Poisson response

For independent Poisson-distributed counts (y_i) with fitted means (\hat{\mu}_i), the deviance is

[ D

2\sum_{i=1}^{n} \left[ y_i\log\left(\frac{y_i}{\hat{\mu}_i}\right)

(y_i-\hat{\mu}_i) \right]. ]

The expression (y_i\log(y_i/\hat{\mu}_i)) is defined as zero when (y_i=0), by continuity. Unlike the unweighted squared residual, each contribution reflects the asymmetric likelihood geometry of count data. A fitted value below an observed positive count does not produce the same contribution as an equally sized difference in the opposite direction.

The related Pearson statistic is

[ X^2

\sum_{i=1}^{n} \frac{(y_i-\hat{\mu}_i)^2}{\hat{\mu}_i}. ]

The deviance and Pearson statistic are asymptotically similar when fitted means are sufficiently regular, but their finite-sample values differ because the Pearson statistic is obtained from a quadratic approximation.

Binomial response

Suppose (y_i) successes are observed in (m_i) trials and the fitted expected number of successes is (\hat{\mu}_i=m_i\hat{p}_i). The binomial deviance is

[ D

2\sum_{i=1}^{n} \left[ y_i\log\left(\frac{y_i}{\hat{\mu}_i}\right) + (m_i-y_i) \log\left( \frac{m_i-y_i}{m_i-\hat{\mu}_i} \right) \right]. ]

Terms with zero numerators are assigned their limiting value of zero. For logistic regression, this quantity compares the fitted probabilities with probabilities that exactly reproduce each response group.

With individual Bernoulli observations, the saturated model assigns probability one to every observed outcome. Residual deviance remains defined, but its use as a direct absolute goodness-of-fit statistic is restricted because each observation supplies only one binary outcome and the usual grouped-count approximation is absent.

Null and residual deviance

The null deviance compares the saturated model with a designated null model. In a regression containing an intercept, the null model ordinarily assigns a common mean on the response scale after accounting for any fixed exposure or offset built into the model.

The residual deviance compares the saturated model with the fitted model containing the selected predictors. Their difference is

[ \Delta D

D_{\text{null}}-D_{\text{residual}}

2\left[ \ell(\hat{\theta}_{\text{fitted}};y)

\ell(\hat{\theta}_{\text{null}};y) \right]. ]

Thus, the reduction in deviance is exactly twice the increase in maximized log-likelihood. The same identity applies to any pair of nested models when the larger model contains the smaller model as a parameter restriction.

Under standard regularity conditions, the deviance difference converges in distribution to a chi-squared distribution. Its degrees of freedom equal the difference in the numbers of freely estimated parameters. Boundary parameters, unidentified components, and certain sparse-data configurations violate the ordinary approximation and produce different limiting distributions.

Deviance residuals

An individual deviance contribution can be converted into a signed residual. For an observation with response (y_i), fitted mean (\hat{\mu}_i), and deviance contribution (d_i), the deviance residual is

[ r_i^{D}

\operatorname{sign}(y_i-\hat{\mu}_i)\sqrt{d_i}. ]

The residuals satisfy

[ \sum_{i=1}^{n}(r_i^{D})^2=D. ]

The sign preserves the direction of the response discrepancy, while the magnitude reflects its likelihood contribution. Deviance residuals therefore place non-Gaussian discrepancies on a scale analogous to ordinary residuals, but they are not generally independent or identically distributed after parameter estimation.

Leverage affects the distribution of residuals because fitted values depend unequally on the observations. Standardized forms incorporate the diagonal elements of the hat matrix or their generalized-linear-model analogues. Such adjustments alter residual diagnostics without changing the model's total deviance.

Interpretation and asymptotic distribution

Deviance is a relative likelihood quantity rather than a universal goodness-of-fit scale. Its value depends on the response distribution, the sampling unit, and the saturated model used as the reference. Splitting one grouped observation into several observations can preserve the underlying data while changing the form of the saturated model and its associated degrees of freedom.

For nested models (M_0\subset M_1), the difference

[ D(M_0)-D(M_1) ]

is the likelihood-ratio test statistic. Under regularity conditions and a true smaller model, this difference has an asymptotic chi-squared distribution. The approximation concerns the difference between nested deviances more directly than it concerns the absolute residual deviance.

An absolute residual deviance is often compared with residual degrees of freedom in Poisson and grouped binomial models. That comparison relies on adequate expected counts, correct specification of the variance structure, and a saturated model whose dimension grows in an appropriate manner. Overdispersion produces residual variation exceeding that prescribed by the nominal distribution, while underdispersion produces the reverse relationship.

For Poisson and multinomial likelihoods, deviance also has an information-theoretic interpretation. After normalization appropriate to the sampling scheme, it is proportional to a Kullback–Leibler divergence from the empirical distribution represented by the saturated model to the fitted distribution. This correspondence explains why deviance is nonnegative and why it vanishes when fitted and empirical distributions coincide.

Relation to model selection

Because deviance is derived from maximized likelihood, it forms part of several likelihood-based model-selection criteria. The Akaike information criterion combines a deviance-equivalent fit term with a penalty proportional to the number of estimated parameters. The Bayesian information criterion uses a penalty that also depends on sample size.

These criteria differ from a pure deviance comparison because they evaluate non-nested models and explicitly account for model dimension. Raw deviance cannot by itself compare models based on different response likelihoods when their likelihood normalizations are not commensurable.

In penalized likelihood, the optimized objective includes a penalty in addition to the log-likelihood. Reported deviance ordinarily remains the unpenalized likelihood discrepancy evaluated at the penalized estimate, so it does not equal the full objective minimized during estimation.