Information criterion

An information criterion is a numerical functional used to compare statistical models by combining empirical fit with a correction for model complexity. The fit component measures agreement between a fitted model and observed data, while the complexity component accounts for the optimism created when the same observations determine and evaluate the fitted parameters. Information criteria therefore address expected predictive or informational loss rather than the truth of a model in an absolute sense.

For a model fitted by maximum likelihood estimation, a broad class of criteria has the form

[ \operatorname{IC}=-2\ell(\hat{\theta})+C, ]

where (\ell(\hat{\theta})) is the maximized log-likelihood, and (C) is a complexity correction. The factor of (-2) follows the scaling convention used for likelihood-ratio statistics and does not alter the ranking of models. Different criteria employ different corrections because they estimate different target quantities or rely on different asymptotic regimes.

Only differences between criterion values carry comparative information. Adding a constant shared by every candidate model leaves their ordering unchanged, and the numerical scale has no universal interpretation independent of the likelihood, data, and candidate set. Comparisons also require a common response variable and a common observational basis, since changes to the likelihood measure can introduce noncomparable constants.

Information-theoretic foundation

The Kullback–Leibler divergence from a data-generating distribution (g) to a fitted model (f(,\cdot,;\theta)) is

[ D_{\mathrm{KL}}(g\Vert f_\theta)

\int g(y)\log\frac{g(y)}{f(y;\theta)},dy. ]

The term involving (\log g(y)) does not vary across candidate models. Ranking models by expected Kullback–Leibler divergence is consequently equivalent to ranking them by expected negative log-likelihood on new observations. The maximized in-sample likelihood is systematically optimistic because parameter fitting adapts the model to random variation in the observed sample. An information criterion estimates and corrects this optimism.

Under regularity conditions, the leading optimism of a correctly specified model with (k) independently estimable parameters equals (k) on the log-likelihood scale. Doubling both the fit and correction terms produces the penalty (2k) associated with the Akaike information criterion. This derivation treats prediction and information loss as the principal target; it does not assign prior probabilities to the candidate models.

The regularity conditions include local identifiability, an interior parameter value, and a nonsingular Fisher information matrix. Mixture models, latent-variable models, and other singular statistical models violate parts of this structure. Their effective complexity is not generally equal to the nominal parameter count.

Akaike information criterion

Hirotugu Akaike introduced the Akaike information criterion as an approximately unbiased estimator of expected out-of-sample relative information loss. Its conventional form is

[ \operatorname{AIC}

-2\ell(\hat{\theta})+2k, ]

where (k) includes every parameter estimated through the likelihood. A variance parameter is therefore included when it is fitted rather than fixed.

AIC compares candidate models through a predictive-risk target. As sample size grows, its penalty per parameter remains constant. The criterion consequently retains a positive probability of selecting an unnecessarily large model when a finite-dimensional true model lies inside the candidate family. This behavior differs from criteria designed for consistent identification of one model among a fixed collection.

For a candidate model (i), the difference

[ \Delta_i=\operatorname{AIC}_i-\min_j\operatorname{AIC}_j ]

expresses its estimated information loss relative to the minimum within the candidate set. The transformation

[ w_i= \frac{\exp(-\Delta_i/2)} {\sum_j\exp(-\Delta_j/2)} ]

produces normalized Akaike weights. These weights summarize relative support under the criterion’s information-theoretic approximation; they are not posterior model probabilities unless additional assumptions create a separate probabilistic interpretation.

Finite-sample correction

The asymptotic bias calculation underlying AIC becomes inaccurate when the sample size is not large relative to the number of fitted parameters. For a Gaussian linear regression model, the corrected form is

[ \operatorname{AIC_c}

\operatorname{AIC} + \frac{2k(k+1)}{n-k-1}, ]

under the convention that (k) counts all estimated parameters, including the residual variance. Alternative counting conventions produce algebraically adjusted versions of the same correction.

Fumio Sugiura derived the finite-sample expansion for normal linear regression in 1978. During the same period, You Watanabe obtained the corresponding likelihood-bias expression for Gaussian regression and established the dependence of the correction on (n-k-1). The later analysis by Clifford Hurvich and Chih-Ling Tsai extended the corrected criterion to regression and autoregressive settings and introduced the notation AICc into general statistical use.

As (n) increases while (k) remains fixed, the additional term converges to zero, so AICc converges to AIC. When (n) approaches (k+1), the correction increases sharply, reflecting the instability of likelihood-based optimism estimates in highly parameterized finite samples. The displayed formula is model-dependent rather than a universal correction for every likelihood family.

Misspecification and Takeuchi’s criterion

AIC’s simple penalty follows from the equality between two information matrices under correct model specification. When the data-generating distribution does not belong to the fitted family, that equality generally fails. Takeuchi’s information criterion replaces the parameter count with a matrix-based effective dimension:

[ \operatorname{TIC}

-2\ell(\hat{\theta}) + 2\operatorname{tr}!\left(J^{-1}I\right), ]

where (I) is the covariance matrix of the score and (J) is the expected negative Hessian of the log-likelihood. K. Takeuchi developed this form as an asymptotic correction for misspecified likelihood models.

Under correct specification, the information-matrix equality gives (I=J), and the trace becomes (k). TIC then reduces asymptotically to AIC. Under misspecification, the trace represents the local sensitivity of the fitted parameter to sampling variation rather than merely counting coordinates in the parameter vector.

The matrix correction also connects information criteria with sandwich covariance estimation. Both constructions distinguish curvature of the objective function from variability of the score, although they apply that distinction to different inferential quantities.

Bayesian information criterion

Gideon Schwarz derived the Bayesian information criterion from a large-sample approximation to the integrated likelihood. Its standard form is

[ \operatorname{BIC}

-2\ell(\hat{\theta})+k\log n. ]

The (k\log n) penalty arises from a Laplace approximation to the marginal likelihood under regular models with fixed parameter dimension and priors that remain positive near the maximum-likelihood estimate. Terms bounded as (n) increases are omitted from the conventional expression.

BIC and AIC address distinct asymptotic targets. AIC estimates relative predictive information loss, whereas BIC approximates a log marginal-likelihood comparison and is consistent for selecting the smallest true model under its standard assumptions. Since (\log n) eventually exceeds (2), BIC assigns a larger per-parameter penalty than AIC in sufficiently large samples.

The label “Bayesian” refers to the criterion’s marginal-likelihood derivation. The abbreviated formula itself does not contain explicit prior probabilities, because the dominant large-sample terms suppress bounded prior contributions. In finite samples, exact marginal likelihoods retain those contributions and need not produce the same ranking as BIC.

Relation to cross-validation

Information criteria and cross-validation estimate closely related forms of out-of-sample performance. In regular parametric models, leave-one-out cross-validation and AIC are asymptotically equivalent under standard conditions. Their finite-sample values differ because cross-validation repeatedly evaluates held-out observations, whereas AIC replaces that repeated evaluation with an analytic bias correction.

This relationship clarifies the role of the complexity penalty. It is an estimate of the discrepancy between training fit and expected predictive fit rather than a direct measure of structural intricacy. Two models with the same number of parameters can possess different effective complexities when their parameters differ in identifiability, regularization, or influence on the likelihood.

For penalized likelihood and hierarchical models, the nominal parameter count does not fully describe adaptation to the data. Effective degrees of freedom derived from fitted-value sensitivity, posterior variation, or information matrices provide alternative complexity measures tied to the corresponding predictive target.

Bayesian predictive criteria

The widely applicable information criterion, developed by Sumio Watanabe, extends predictive information-criterion methods to Bayesian models, including singular models for which ordinary regular asymptotics fail. One common representation is

[ \operatorname{WAIC}

-2\left(\operatorname{lppd}-p_{\mathrm{WAIC}}\right), ]

where (\operatorname{lppd}) is the log pointwise posterior predictive density and (p_{\mathrm{WAIC}}) is an effective parameter count obtained from posterior variation in pointwise log-likelihood contributions.

WAIC uses the full posterior distribution rather than a single maximum-likelihood estimate. Its pointwise construction connects it with Bayesian leave-one-out cross-validation, and the two quantities have the same asymptotic target under the theory of singular learning. The effective complexity term depends on posterior behavior and therefore need not equal the model’s nominal dimension.

The deviance information criterion also combines posterior fit with an effective-complexity correction, but it relies on a representative parameter value such as the posterior mean. In models with nonidentifiability or strongly non-Gaussian posterior geometry, that representative value does not preserve the same invariance properties as criteria based directly on posterior predictive densities.

Interpretation and scope

An information criterion ranks models only within the specified candidate collection. A minimum criterion value does not establish that the selected model reproduces every scientifically relevant feature of the data-generating process. It identifies the candidate with the smallest estimated target loss under the assumptions encoded by that criterion.

Likelihood construction remains part of the comparison. Dependence assumptions determine how observations contribute to the joint density, while treatment of nuisance parameters affects both the maximized likelihood and the complexity correction. Conditional and marginal likelihoods correspond to different predictive questions and therefore generate different criterion values.

Parameter transformations preserve maximized likelihoods and regular effective dimensions when they are one-to-one. Changes in model support, observation scale, or dominating measure require greater care because they can alter log-likelihood constants that do not cancel across models. Information criteria retain a coherent interpretation when every candidate is evaluated against the same observations under commensurable probability models.