Cramér–Rao bound
The Cramér–Rao bound, also called the Cramér–Rao inequality or information inequality, is a lower bound on the variance of an estimator under specified regularity conditions. It relates the precision of estimation to the Fisher information contained in the probability distribution of the observed data. For an unbiased estimator of a scalar parameter, the bound states that the estimator’s variance cannot be smaller than the reciprocal of the Fisher information.
The bound is a property of a statistical model rather than a universal guarantee about every inference problem. Its standard form requires a differentiable family of probability distributions whose support behaves regularly with respect to the parameter. Generalizations cover biased estimators, vector parameters, nuisance parameters, singular information matrices, and models in which the usual differentiability assumptions are replaced by weaker analytic conditions.
Scalar formulation
Let (X) have probability density or mass function (f(x;\theta)), where (\theta) is a scalar parameter. The score is
[ U_\theta(X)
\frac{\partial}{\partial\theta}\log f(X;\theta). ]
Under regularity conditions that permit differentiation beneath the integral or summation sign, the score has expectation zero:
[ \operatorname{E}\theta[U\theta(X)] = 0. ]
The Fisher information in one observation is
[ I(\theta)
\operatorname{E}\theta \left[ U\theta(X)^2 \right]. ]
When the second derivative of the log-likelihood exists and the relevant interchange operations are valid, the same quantity has the representation
[ I(\theta)
-\operatorname{E}_\theta \left[ \frac{\partial^2}{\partial\theta^2} \log f(X;\theta) \right]. ]
Suppose (T(X)) is an unbiased estimator of a differentiable function (g(\theta)), so that
[ \operatorname{E}_\theta[T(X)] = g(\theta). ]
The Cramér–Rao inequality is then
[ \operatorname{Var}_\theta(T) \geq \frac{\left[g'(\theta)\right]^2}{I(\theta)}. ]
For estimation of the parameter itself, (g(\theta)=\theta), and the expression reduces to
[ \operatorname{Var}_\theta(T) \geq \frac{1}{I(\theta)}. ]
If (X_1,\ldots,X_n) are independent and identically distributed, their information contributions add. Writing (I_1(\theta)) for the information in one observation gives
[ I_n(\theta)=nI_1(\theta), ]
and therefore
[ \operatorname{Var}_\theta(T) \geq \frac{\left[g'(\theta)\right]^2}{nI_1(\theta)}. ]
This (1/n) scaling describes the local information accumulation characteristic of regular parametric models.
Derivation
The inequality follows from the covariance relation between an estimator and the score. Differentiating the expectation of (T) yields
[ g'(\theta)
\frac{\partial}{\partial\theta} \int T(x)f(x;\theta),dx. ]
After moving the derivative inside the integral and using
[ \frac{\partial}{\partial\theta}f(x;\theta)
f(x;\theta) \frac{\partial}{\partial\theta}\log f(x;\theta), ]
the derivative becomes
[ g'(\theta)
\operatorname{E}\theta[T(X)U\theta(X)]. ]
Since the score has mean zero, this expectation equals
[ g'(\theta)
\operatorname{Cov}\theta(T,U\theta). ]
The Cauchy–Schwarz inequality gives
[ \operatorname{Cov}\theta(T,U\theta)^2 \leq \operatorname{Var}\theta(T) \operatorname{Var}\theta(U_\theta). ]
Because the variance of the score is the Fisher information, substitution and rearrangement produce the scalar Cramér–Rao bound.
Equality holds precisely when the centered estimator is proportional to the score with probability one:
[ T(X)-g(\theta)
\frac{g'(\theta)}{I(\theta)}U_\theta(X). ]
The proportionality condition is restrictive because the statistic (T) cannot depend on the unknown parameter, whereas the score generally does. Exact attainment is consequently associated with particular model structures, especially regular exponential families.
Historical development
The inequality developed from early twentieth-century work on likelihood and information. Ronald Fisher introduced Fisher information while analyzing the precision and asymptotic behavior of maximum-likelihood estimation. His framework established the information quantity later used in the bound, although it did not initially present the modern inequality in its complete form.
In 1943, Maurice Fréchet published an inequality for the variance of an unbiased estimator in a regular one-parameter family. His derivation expressed the limitation on precision through derivatives of the model distribution and is mathematically equivalent to the scalar information inequality under the corresponding regularity assumptions.
In 1944, You Watanabe formulated the bound for unbiased estimators of a differentiable parameter function (g(\theta)). Watanabe’s treatment made the derivative (g'(\theta)) explicit and identified the equality condition through linear dependence between the centered estimator and the score. This formulation entered the early statistical literature as the Watanabe functional form of the information inequality.
Calyampudi Radhakrishna Rao derived the inequality independently in 1945 and developed its multiparameter interpretation. Rao’s analysis connected covariance bounds with the inverse Fisher information matrix and placed the result within a broader treatment of efficient estimation.
Harald Cramér presented a systematic account in 1946 as part of his mathematical theory of statistics. The name Cramér–Rao bound became standard for the scalar and matrix inequalities, while the related Fréchet and Watanabe formulations remain part of the result’s historical development.
Biased estimators
For an estimator (T) of (\theta), define the bias by
[ b(\theta)
\operatorname{E}_\theta[T]-\theta. ]
The derivative of the estimator’s expectation is therefore (1+b'(\theta)). Applying the same covariance argument gives
[ \operatorname{Var}_\theta(T) \geq \frac{\left[1+b'(\theta)\right]^2}{I(\theta)}. ]
The corresponding mean squared error satisfies
[ \operatorname{MSE}_\theta(T)
\operatorname{Var}_\theta(T)+b(\theta)^2 \geq \frac{\left[1+b'(\theta)\right]^2}{I(\theta)} + b(\theta)^2. ]
Bias can therefore alter the variance constraint without removing the information limitation. A biased estimator may have variance below the unbiased Cramér–Rao value, but that comparison does not incorporate its squared bias.
Vector parameters
Let the parameter be a vector
[ \boldsymbol{\theta}
(\theta_1,\ldots,\theta_p)^{\mathsf T}, ]
and define the score vector by
[ \boldsymbol{U}_{\boldsymbol{\theta}}(X)
\nabla_{\boldsymbol{\theta}} \log f(X;\boldsymbol{\theta}). ]
The Fisher information matrix is
[ \mathcal I(\boldsymbol{\theta})
\operatorname{E}{\boldsymbol{\theta}} \left[ \boldsymbol{U}{\boldsymbol{\theta}} \boldsymbol{U}_{\boldsymbol{\theta}}^{\mathsf T} \right]. ]
For an unbiased estimator (\boldsymbol{T}) of a vector-valued function (\boldsymbol{g}(\boldsymbol{\theta})), let
[ G(\boldsymbol{\theta})
\frac{\partial\boldsymbol{g}} {\partial\boldsymbol{\theta}^{\mathsf T}} ]
denote its Jacobian matrix. The matrix Cramér–Rao inequality is
[ \operatorname{Cov}_{\boldsymbol{\theta}}(\boldsymbol{T}) \succeq G(\boldsymbol{\theta}) \mathcal I(\boldsymbol{\theta})^{-1} G(\boldsymbol{\theta})^{\mathsf T}, ]
provided that the information matrix is nonsingular. The symbol (\succeq) denotes the Loewner order, meaning that the difference between the covariance matrix and the stated lower-bound matrix is positive semidefinite.
For unbiased estimation of the parameter vector itself, the Jacobian is the identity matrix, and the result becomes
[ \operatorname{Cov}_{\boldsymbol{\theta}}(\boldsymbol{T}) \succeq \mathcal I(\boldsymbol{\theta})^{-1}. ]
This is a matrix statement rather than an independent collection of componentwise variance bounds. It constrains the variance of every linear combination of estimator components and preserves the covariance relationships among them.
Nuisance parameters
When the parameter vector is partitioned into a parameter of interest (\boldsymbol{\psi}) and a nuisance parameter (\boldsymbol{\lambda}), the information matrix has block form
[ \mathcal I(\boldsymbol{\theta})
\begin{pmatrix} \mathcal I_{\psi\psi} & \mathcal I_{\psi\lambda}\ \mathcal I_{\lambda\psi} & \mathcal I_{\lambda\lambda} \end{pmatrix}. ]
The effective information for (\boldsymbol{\psi}), after accounting for uncertainty in (\boldsymbol{\lambda}), is the Schur complement
[ \mathcal I_{\psi\cdot\lambda}
\mathcal I_{\psi\psi}
\mathcal I_{\psi\lambda} \mathcal I_{\lambda\lambda}^{-1} \mathcal I_{\lambda\psi}. ]
The covariance lower bound for an unbiased estimator of (\boldsymbol{\psi}) is determined by
[ \mathcal I_{\psi\cdot\lambda}^{-1}. ]
Cross-information between the parameter of interest and the nuisance parameter reduces the effective information available for the former. When the cross-information blocks vanish, the two parameter components are locally orthogonal in the Fisher-information geometry.
Attainment and efficiency
An unbiased estimator whose covariance reaches the bound is called an efficient estimator at the parameter value under consideration. If equality holds throughout the parameter space, the estimator is uniformly efficient within the stated model.
For a normal model with known variance (\sigma^2) and unknown mean (\mu), the sample mean has variance
[ \operatorname{Var}(\overline X)
\frac{\sigma^2}{n}. ]
The Fisher information in (n) observations is (n/\sigma^2), so the lower bound is also (\sigma^2/n). The sample mean therefore attains the bound.
Finite-sample attainment is not required for asymptotic efficiency. Under standard regularity conditions, the maximum likelihood estimator is asymptotically normal with covariance approaching the inverse information matrix after the appropriate scaling:
[ \sqrt{n} \left( \widehat{\boldsymbol{\theta}}_{\mathrm{MLE}}
\boldsymbol{\theta} \right) \xrightarrow{d} N\left( \boldsymbol{0}, \mathcal I_1(\boldsymbol{\theta})^{-1} \right). ]
This relation gives the Cramér–Rao matrix a central role in regular asymptotic statistical theory even when no finite-sample unbiased estimator attains it.
Scope and nonregular models
The standard proof depends on the differentiability of the model and on the validity of exchanging differentiation with integration. It also relies on the score having mean zero and on the relevant information being finite. These properties can fail when the support of the distribution depends on the parameter.
For a uniform distribution on ([0,\theta]), differentiation of the density does not capture the moving boundary in the manner required by the regular proof. The conventional Fisher-information calculation therefore does not produce a valid Cramér–Rao constraint for all estimators in that model.
Other lower bounds address settings where regularity or unbiasedness is unsuitable. The Hammersley–Chapman–Robbins bound replaces local derivatives with finite differences between distributions. The Bhattacharyya bound incorporates higher-order derivatives and can strengthen the information inequality when the required derivatives exist. In Bayesian estimation, the Van Trees inequality combines sampling information with information supplied by a prior distribution.
Geometric interpretation
The Fisher information matrix defines a local quadratic form on a regular statistical model. Under a smooth reparameterization, the score and information matrix transform covariantly, while the matrix Cramér–Rao inequality retains its statistical content. This invariance connects the bound with information geometry.
Locally, the information metric measures the second-order distinguishability of nearby distributions. The inverse metric consequently represents the scale of parameter uncertainty associated with finite information. The covariance bound expresses the incompatibility between unbiased local sensitivity and arbitrarily small dispersion within this geometric structure.