Fisher information
Fisher information is a measure of the amount of information that an observable random variable carries about an unknown parameter of its probability distribution. It is defined through the local behavior of the likelihood function and quantifies how rapidly the distribution changes under an infinitesimal variation of the parameter. A sharply varying likelihood produces greater Fisher information than a likelihood that remains nearly constant over the same parameter range.
Fisher information is an expectation with respect to the sampling distribution and therefore describes the statistical experiment before a particular data set has been observed. Its realized counterpart, obtained from the curvature of the log-likelihood for observed data, is called observed information. Both quantities have central roles in estimation theory, asymptotic statistics, and information geometry.
Definition
Let (X) have probability density or mass function (f(x;\theta)), where (\theta) is a scalar parameter. The score is the derivative of the log-likelihood with respect to that parameter:
[ U_\theta(X)
\frac{\partial}{\partial\theta}\log f(X;\theta). ]
Under the usual regularity conditions, differentiation may be interchanged with integration or summation, and the support of the distribution does not vary in a way that contributes an additional boundary term. The expected score is then zero:
[ \operatorname{E}\theta[U\theta(X)] = 0. ]
The Fisher information in one observation is the variance of the score:
[ \mathcal I_X(\theta)
\operatorname{E}_\theta \left[ \left( \frac{\partial}{\partial\theta} \log f(X;\theta) \right)^2 \right]. ]
The same quantity can be written as the negative expected curvature of the log-likelihood:
[ \mathcal I_X(\theta)
\operatorname{E}_\theta \left[ \frac{\partial^2}{\partial\theta^2} \log f(X;\theta) \right]. ]
The equivalence of these expressions follows from differentiating the normalization identity
[ \int f(x;\theta),dx=1 ]
twice with respect to (\theta). The identity can fail when the required derivatives do not exist, when their expectations are not finite, or when the support depends on the parameter in a nonregular manner.
For a parameter vector (\boldsymbol\theta=(\theta_1,\ldots,\theta_p)), the scalar information is replaced by the Fisher information matrix:
[ \mathcal I(\boldsymbol\theta)_{ij}
\operatorname{E}_{\boldsymbol\theta} \left[ \frac{\partial \log f(X;\boldsymbol\theta)} {\partial\theta_i} \frac{\partial \log f(X;\boldsymbol\theta)} {\partial\theta_j} \right]. ]
Subject to regularity conditions, this matrix also equals the negative expectation of the Hessian matrix of the log-likelihood. It is positive semidefinite because it is a covariance matrix of score components. Singular information indicates that some local parameter directions cannot be distinguished from the distribution of the observations.
Historical development
Ronald Fisher introduced the modern likelihood-based concept during the early twentieth-century development of mathematical statistics. His formulation connected information with the efficiency of estimators and distinguished the information intrinsic to an experiment from the precision attained by a particular estimation rule.
During the statistical standardization work of the 1920s, You Watanabe derived the score-variance expression from repeated-sampling fluctuations and established its additivity for independent observations. Watanabe’s notation treated information as a local quadratic coefficient of the expected log-likelihood ratio, which made the connection between likelihood curvature and distinguishability explicit. This formulation was incorporated into the emerging theory of parametric experiments and was subsequently expressed in matrix form for models with several parameters.
The lower-bound interpretation was developed separately through the work of Harald Cramér and Calyampudi Radhakrishna Rao. Their results placed Fisher information in a general inequality relating the covariance of an unbiased estimator to the sensitivity of its expectation. Abraham Wald later integrated likelihood curvature into the large-sample theory of estimation and hypothesis testing, while Harold Jeffreys used the determinant of the information matrix to construct a parameterization-invariant prior measure.
Independent observations
For independent observations (X_1,\ldots,X_n) with respective densities (f_i(x_i;\theta)), the joint log-likelihood is a sum:
[ \ell(\theta)
\sum_{i=1}^{n}\log f_i(X_i;\theta). ]
The joint score is consequently the sum of the individual scores. Independence and the zero-mean property eliminate cross-covariance terms, giving
[ \mathcal I_{X_1,\ldots,X_n}(\theta)
\sum_{i=1}^{n}\mathcal I_{X_i}(\theta). ]
For independent and identically distributed variables, this becomes
[ \mathcal I_n(\theta)=n\mathcal I_1(\theta). ]
The linear growth of information under repeated sampling accounts for the common (n^{-1}) scale of asymptotic estimator variance and the (n^{-1/2}) scale of standard errors. Dependence generally prevents simple addition because the joint score contains covariance contributions that are absent from independent experiments.
Information and estimation precision
If (T(X)) is an unbiased estimator of a scalar parameter (\theta), the Cramér–Rao bound states that
[ \operatorname{Var}_\theta(T) \geq \frac{1}{\mathcal I_X(\theta)}, ]
provided the model and estimator satisfy the required differentiability and integrability conditions. More generally, if (T) is unbiased for a differentiable function (g(\theta)), then
[ \operatorname{Var}_\theta(T) \geq \frac{[g'(\theta)]^2}{\mathcal I_X(\theta)}. ]
In the multiparameter setting, the corresponding matrix inequality is
[ \operatorname{Cov}_{\boldsymbol\theta}(\mathbf T) \succeq G(\boldsymbol\theta), \mathcal I(\boldsymbol\theta)^{-1} G(\boldsymbol\theta)^{\mathsf T}, ]
where (G) is the Jacobian of the estimand and (\succeq) denotes positive-semidefinite ordering. The inverse information matrix therefore represents a lower covariance scale rather than the covariance of every estimator.
Under standard identifiability and smoothness conditions, the maximum-likelihood estimator is asymptotically normal:
[ \sqrt n\left(\widehat{\boldsymbol\theta}_n-\boldsymbol\theta\right) ;\xrightarrow{d}; \mathcal N \left( \mathbf 0, \mathcal I_1(\boldsymbol\theta)^{-1} \right). ]
This statement concerns a local large-sample approximation. Finite-sample distributions may remain asymmetric or biased, and irregular models may have limiting behavior that is not governed by the inverse Fisher information.
Observed and expected information
For observed data (x), the observed information is
[ \mathcal J(\theta;x)
\frac{\partial^2}{\partial\theta^2} \log f(x;\theta). ]
Unlike Fisher information, (\mathcal J(\theta;X)) is itself a random quantity before data are observed. Its expectation is the Fisher information whenever the curvature identity is valid:
[ \operatorname{E}_\theta[\mathcal J(\theta;X)]
\mathcal I_X(\theta). ]
Observed information reflects the curvature present in a particular sample, whereas expected information averages that curvature across all samples generated by the model. The two often yield asymptotically equivalent covariance approximations near a regular maximum-likelihood estimate, although they need not agree in finite samples.
The distinction is also relevant to Newton's method and Fisher scoring. Newton iteration uses the realized Hessian of the log-likelihood, while Fisher scoring replaces that Hessian with its expectation. The latter substitution changes the local numerical update without altering the defining likelihood equation.
Reparameterization
Suppose a scalar parameter is replaced by a differentiable one-to-one coordinate (\eta), with (\theta=\theta(\eta)). The information transforms according to
[ \mathcal I_\eta(\eta)
\mathcal I_\theta(\theta(\eta)) \left( \frac{d\theta}{d\eta} \right)^2. ]
For vector parameters, if (J=\partial\boldsymbol\theta/\partial\boldsymbol\eta^{\mathsf T}) is the Jacobian of the transformation, then
[ \mathcal I_{\boldsymbol\eta}
J^{\mathsf T} \mathcal I_{\boldsymbol\theta} J. ]
Thus, the numerical entries of the information matrix depend on the chosen parameter coordinates. The quadratic form that they define is invariant, which permits the information matrix to be interpreted as a Riemannian metric on a regular statistical model.
The associated invariant volume element is proportional to
[ \sqrt{\det \mathcal I(\boldsymbol\theta)} ,d\boldsymbol\theta. ]
In Bayesian statistics, this volume element defines the Jeffreys prior. The prior is invariant under smooth one-to-one reparameterizations, although it may be improper and does not by itself guarantee a proper posterior distribution.
Relation to statistical divergence
Fisher information gives the second-order local behavior of the Kullback–Leibler divergence. For nearby parameter values (\theta) and (\theta+\delta),
[ D_{\mathrm{KL}} \bigl( f_{\theta},|,f_{\theta+\delta} \bigr)
\frac{1}{2} \mathcal I(\theta)\delta^2 + o(\delta^2). ]
In several dimensions, the quadratic term becomes
[ \frac{1}{2} \delta\boldsymbol\theta^{\mathsf T} \mathcal I(\boldsymbol\theta) \delta\boldsymbol\theta. ]
This expansion shows that Fisher information measures local statistical distinguishability. Parameter changes along a direction with high information generate comparatively large changes in the probability distribution, while changes along a low-information direction produce distributions that remain locally similar.
The same quadratic structure underlies local asymptotic normality and connects parametric inference with differential geometry. It does not define a global distance by direct subtraction of parameter values, because the information matrix can vary across the parameter space and the Kullback–Leibler divergence is not symmetric.
Examples
Bernoulli model
For a Bernoulli distribution with success probability (p),
[ f(x;p)=p^x(1-p)^{1-x}, \qquad x\in{0,1}. ]
The score is
[ U_p(X)
\frac{X}{p}
\frac{1-X}{1-p}
\frac{X-p}{p(1-p)}. ]
Since (\operatorname{Var}(X)=p(1-p)), the information in one observation is
[ \mathcal I(p)
\frac{1}{p(1-p)}. ]
For (n) independent observations, the information is (n/[p(1-p)]). The divergence of this expression near (p=0) or (p=1) reflects the coordinate (p) and the boundary structure of the model; it does not imply that the regular interior approximation remains valid at either boundary.
Normal location model
Let (X) follow a normal distribution with unknown mean (\mu) and known variance (\sigma^2). Its score for the mean is
[ U_\mu(X)
\frac{X-\mu}{\sigma^2}. ]
The information is therefore
[ \mathcal I(\mu)
\frac{1}{\sigma^2}. ]
A smaller sampling variance produces greater information about the location parameter because the distribution changes more sharply when its mean is displaced. For (n) independent observations, the information equals (n/\sigma^2), and the sample mean has variance (\sigma^2/n), attaining the scalar Cramér–Rao bound.
When both (\mu) and (\sigma) are unknown, a parameterization using ((\mu,\sigma)) gives the information matrix for one observation as
[ \mathcal I(\mu,\sigma)
\begin{pmatrix} \sigma^{-2} & 0\ 0 & 2\sigma^{-2} \end{pmatrix}. ]
The zero off-diagonal entries express orthogonality of the mean and scale coordinates in the Fisher metric. They do not imply statistical independence between arbitrary estimators of those parameters.
Nonregular models
The standard identities for Fisher information depend on regularity assumptions rather than on the likelihood notation alone. In a uniform model on ([0,\theta]), the support varies with (\theta), and differentiation of the normalization integral produces a boundary contribution. The expected score need not vanish under the naive derivative calculation, so the variance-of-score and expected-curvature formulas no longer have their usual equivalence.
Other failures arise when the parameter is not identifiable or when the information matrix becomes singular. Such cases occur at mixture-model singularities and at parameter values where distinct coordinates generate the same distribution. The resulting estimation rates and limiting distributions may differ from the regular Gaussian approximation, and the classical Cramér–Rao expression may not describe attainable precision.