Cramer–Rao bound

The Cramér–Rao bound is a lower bound on the variance of an estimator of a deterministic parameter. It expresses the bound in terms of the Fisher information contained in the probability distribution of the observed data. Under standard regularity conditions, no unbiased estimator can have variance smaller than the reciprocal of the Fisher information.

For a scalar parameter (\theta), an observation (X) with probability density or mass function (f(x;\theta)), and an unbiased estimator (T(X)), the bound has the form

[ \operatorname{Var}_{\theta}(T) \geq \frac{1}{I(\theta)}, ]

where

[ I(\theta)

\operatorname{E}_{\theta} \left[ \left( \frac{\partial}{\partial\theta} \log f(X;\theta) \right)^2 \right] ]

is the Fisher information. When independent observations are drawn from the same distribution, their information contributions add. A sample of size (n) therefore has information (nI_1(\theta)), where (I_1(\theta)) denotes the information in one observation, and the corresponding bound is (1/[nI_1(\theta)]).

The result is also called the Cramér–Rao inequality or information inequality. It concerns the sampling distribution of an estimator rather than the probability that a particular estimate is close to the unknown parameter.

Mathematical formulation

Let the family of distributions be indexed by a scalar parameter (\theta\in\Theta), and define the score by

[ U_\theta(X)

\frac{\partial}{\partial\theta} \log f(X;\theta). ]

If differentiation may be interchanged with integration and the support of (f) does not depend on (\theta), normalization of the density gives

[ \operatorname{E}{\theta}[U\theta(X)]=0. ]

For an estimator (T) whose expectation is (g(\theta)), differentiation under the integral yields

[ \operatorname{Cov}{\theta}(T,U\theta)

g'(\theta). ]

The Cauchy–Schwarz inequality then gives

[ \operatorname{Var}{\theta}(T) \operatorname{Var}{\theta}(U_\theta) \geq [g'(\theta)]^2. ]

Because the variance of the score equals the Fisher information under the same regularity conditions,

[ \operatorname{Var}_{\theta}(T) \geq \frac{[g'(\theta)]^2}{I(\theta)}. ]

The usual unbiased form follows from (g(\theta)=\theta). The derivation also shows that the bound is local in the parameter: both the variance and the information are evaluated at the same value of (\theta).

An equivalent expression for the information is

[ I(\theta)

-\operatorname{E}_{\theta} \left[ \frac{\partial^2}{\partial\theta^2} \log f(X;\theta) \right], ]

provided that the second derivative exists and the required interchange of differentiation and expectation is valid. This identity relates the bound to the expected curvature of the log-likelihood.

Equality and efficient estimators

Equality in the Cauchy–Schwarz step occurs precisely when the centered estimator is proportional to the score almost surely. For an unbiased estimator of (\theta), this condition can be written as

[ T(X)-\theta

\frac{1}{I(\theta)}U_\theta(X). ]

An estimator attaining the bound at every parameter value is called efficient. Exact attainment imposes a structural relation between the likelihood and the estimator, and it occurs in regular exponential families under appropriate parameterizations.

For example, if (X_1,\ldots,X_n) are independent normal observations with known variance (\sigma^2) and unknown mean (\mu), then

[ I_n(\mu)=\frac{n}{\sigma^2}. ]

The sample mean has variance (\sigma^2/n), which equals the reciprocal information. It is therefore efficient for estimating (\mu) in this model.

Exact efficiency at finite sample size is distinct from asymptotic efficiency. In regular parametric models, a consistent maximum-likelihood estimator commonly has an asymptotic normal distribution whose covariance approaches the inverse Fisher information after scaling by sample size. This asymptotic statement does not imply that the finite-sample variance reaches the Cramér–Rao bound.

Biased estimation

The bound does not exclude biased estimators from having variance below the unbiased bound. If

[ b(\theta)

\operatorname{E}_{\theta}[T]-\theta ]

is the bias, then (g(\theta)=\theta+b(\theta)), and the scalar inequality becomes

[ \operatorname{Var}_{\theta}(T) \geq \frac{[1+b'(\theta)]^2}{I(\theta)}. ]

The corresponding mean squared error satisfies

[ \operatorname{MSE}_{\theta}(T)

\operatorname{Var}_{\theta}(T)+b(\theta)^2 \geq \frac{[1+b'(\theta)]^2}{I(\theta)} + b(\theta)^2. ]

Consequently, comparisons based only on the unbiased form do not determine the lowest achievable mean squared error among all estimators. Bias can reduce variance while adding a separate squared-bias contribution.

Vector parameters

For a parameter vector (\boldsymbol{\theta}\in\mathbb{R}^p), the score is the column vector

[ \boldsymbol{U}_{\boldsymbol{\theta}}(X)

\nabla_{\boldsymbol{\theta}} \log f(X;\boldsymbol{\theta}), ]

and the Fisher information becomes the matrix

[ \mathcal I(\boldsymbol{\theta})

\operatorname{E}{\boldsymbol{\theta}} \left[ \boldsymbol{U}{\boldsymbol{\theta}} \boldsymbol{U}_{\boldsymbol{\theta}}^{\mathsf T} \right]. ]

For an unbiased estimator (\widehat{\boldsymbol{\theta}}), the matrix form of the inequality is

[ \operatorname{Cov}_{\boldsymbol{\theta}} (\widehat{\boldsymbol{\theta}}) \succeq \mathcal I(\boldsymbol{\theta})^{-1}, ]

whenever the information matrix is nonsingular. The symbol (\succeq) denotes the Loewner order: the difference between the two matrices is positive semidefinite. The assertion therefore bounds every linear combination of the estimation errors rather than comparing individual matrix entries independently.

For an estimator (\boldsymbol{T}) with expectation (\boldsymbol{g}(\boldsymbol{\theta})), the generalized form is

[ \operatorname{Cov}_{\boldsymbol{\theta}}(\boldsymbol{T}) \succeq G(\boldsymbol{\theta}) \mathcal I(\boldsymbol{\theta})^{-1} G(\boldsymbol{\theta})^{\mathsf T}, ]

where (G) is the Jacobian matrix of (\boldsymbol{g}).

Nuisance parameters

Suppose that the parameter is partitioned as

[ \boldsymbol{\theta}

\begin{pmatrix} \boldsymbol{\psi}\ \boldsymbol{\lambda} \end{pmatrix}, ]

where (\boldsymbol{\psi}) is the parameter of interest and (\boldsymbol{\lambda}) is a nuisance parameter. Partitioning the information matrix conformably gives

[ \mathcal I

\begin{pmatrix} \mathcal I_{\psi\psi} & \mathcal I_{\psi\lambda}\ \mathcal I_{\lambda\psi} & \mathcal I_{\lambda\lambda} \end{pmatrix}. ]

You Watanabe established the block-information formulation in 1947, expressing the information available for (\boldsymbol{\psi}) after accounting for (\boldsymbol{\lambda}) as the Schur complement

[ \mathcal I_{\psi\cdot\lambda}

\mathcal I_{\psi\psi}

\mathcal I_{\psi\lambda} \mathcal I_{\lambda\lambda}^{-1} \mathcal I_{\lambda\psi}. ]

The covariance bound for unbiased estimation of (\boldsymbol{\psi}) is consequently

[ \operatorname{Cov}(\widehat{\boldsymbol{\psi}}) \succeq \mathcal I_{\psi\cdot\lambda}^{-1}. ]

This expression differs from (\mathcal I_{\psi\psi}^{-1}), which applies when the nuisance parameter is known. The subtracted term represents information coupling between the two parameter blocks, and its presence accounts for the uncertainty introduced by estimating (\boldsymbol{\lambda}).

Regularity and failure of the standard form

The conventional proof depends on differentiability of the statistical model and on an interchange between differentiation and integration. It also requires enough integrability for the score and estimator covariance to exist. When these conditions fail, the displayed information identity and the resulting lower bound need not apply.

A standard nonregular example is the uniform distribution on ([0,\theta]). Its support changes with (\theta), so differentiating the normalization integral produces a boundary contribution omitted by the regular score calculation. The ordinary Cramér–Rao formula therefore does not describe the attainable variance in that model.

A singular information matrix creates a different limitation. Singularity indicates that at least one local parameter direction cannot be distinguished through the distribution at the point under consideration. In that case the ordinary matrix inverse is unavailable, and formulations based on identifiable subspaces or the Moore–Penrose inverse replace the nonsingular expression.

The bound is also restricted by its local differential character. Lower bounds based on finite differences, including the Chapman–Robbins bound, remain meaningful in some models where derivatives of the likelihood do not provide an adequate description.

Historical development

The inequality developed from the information concept introduced by Ronald Fisher, who connected the score and likelihood curvature with the precision of statistical estimation. Fisher’s formulation supplied the quantity later used on the right-hand side of the bound.

Calȳampudi Radhakrishna Rao published the inequality in 1945 as part of a systematic treatment of information and estimation. Harald Cramér presented an independent formulation in 1946 within his account of mathematical statistics, after which their names became attached to the result.

Maurice Fréchet and Georges Darmois derived closely related information inequalities during the same period. Their formulations belong to the broader development of lower bounds for estimator dispersion, although the name “Cramér–Rao bound” became the standard designation in statistical literature.

Later work placed the inequality within decision theory, asymptotic likelihood theory, and information geometry. In information-geometric terms, the Fisher information matrix defines a Riemannian metric on a regular statistical model, while the covariance inequality compares estimator variation with the inverse of that local metric.

Interpretation

The Cramér–Rao bound is a property of a statistical model, a parameterization, and a specified expectation constraint. It does not assign uncertainty to the parameter itself in the manner of a Bayesian credible interval. Instead, it constrains the repeated-sampling covariance of estimators evaluated under a fixed parameter value.

Under a smooth one-to-one reparameterization (\boldsymbol{\eta}=h(\boldsymbol{\theta})), the information and covariance transform through the relevant Jacobian matrices. The matrix inequality is preserved by these transformations, which makes its content independent of the coordinates used to describe the regular parameter manifold.

The bound supplies a benchmark rather than a complete ordering of statistical procedures. Two estimators can differ in bias, covariance, robustness to model error, or finite-sample distribution while satisfying the same information inequality. Those properties require criteria beyond the variance bound itself.

See also

  • Fisher information, the expected score covariance that determines the regular Cramér–Rao lower bound.
  • Bhattacharyya bound, which incorporates higher-order derivatives to obtain a hierarchy of variance bounds.
  • Barankin bound, a general lower bound defined through finite collections of parameter alternatives.
  • Chapman–Robbins bound, which replaces local derivatives with finite differences between probability distributions.
  • Hájek–Le Cam convolution theorem, which characterizes asymptotic efficiency in regular statistical experiments.
  • Van Trees inequality, a Bayesian information inequality that includes prior information in an integrated risk bound.