Fisher scoring

Fisher scoring is an iterative method for estimating the parameters of a statistical model by maximizing its likelihood function. The method replaces the observed curvature used by the Newton–Raphson method with the expected curvature represented by the Fisher information. Under standard regularity conditions, the two curvatures become asymptotically equivalent near the true parameter value, although their behavior can differ substantially during early iterations.

The method emerged from Ronald Fisher's development of likelihood theory and statistical information during the 1920s. Its modern matrix form applies to scalar and vector parameters, and it provides the computational basis for several fitting methods used in generalized linear models, contingency-table analysis, and maximum likelihood estimation.

Mathematical formulation

Let (Y) have probability density or mass function (f(y;\theta)), where the parameter (\theta) belongs to a subset of (\mathbb{R}^p). For observed data (y), the log-likelihood is

[ \ell(\theta;y)=\log f(y;\theta). ]

The score vector is the gradient of the log-likelihood,

[ U(\theta)

\frac{\partial \ell(\theta;y)}{\partial\theta}. ]

A maximum-likelihood estimate ordinarily satisfies the likelihood equation

[ U(\widehat{\theta})=0. ]

The observed information matrix is the negative Hessian of the log-likelihood,

[ J(\theta)

-\frac{\partial^2\ell(\theta;y)} {\partial\theta,\partial\theta^{\mathsf T}}. ]

By contrast, the Fisher information matrix is the expectation of the observed information under the model,

[ I(\theta)

\operatorname{E}_{\theta}!\left[J(\theta)\right]. ]

When differentiation and integration can be interchanged and the model satisfies the relevant regularity conditions, the information matrix also has the representation

[ I(\theta)

\operatorname{E}_{\theta} !\left[ U(\theta)U(\theta)^{\mathsf T} \right]. ]

Starting from a parameter value (\theta^{(t)}), the Fisher scoring update is

[ \theta^{(t+1)}

\theta^{(t)} + I!\left(\theta^{(t)}\right)^{-1} U!\left(\theta^{(t)}\right). ]

This expression is defined when the information matrix is nonsingular. In constrained or non-identifiable models, the same formal update can require a generalized inverse or a reparameterized information matrix.

Relation to Newton–Raphson iteration

Newton–Raphson maximization uses the local second derivative of the realized log-likelihood. Its corresponding update is

[ \theta^{(t+1)}

\theta^{(t)} + J!\left(\theta^{(t)}\right)^{-1} U!\left(\theta^{(t)}\right). ]

Fisher scoring therefore differs from Newton–Raphson only through its curvature matrix. The observed information (J(\theta)) depends directly on the realized sample, whereas the expected information (I(\theta)) averages that curvature over hypothetical samples generated at (\theta).

Near a regular maximum-likelihood estimate, both matrices generally provide the same first-order asymptotic description. Their finite-sample properties remain distinct because observed curvature can include substantial sample-specific variation. Expected information is positive semidefinite by its score-covariance representation, but this property does not by itself establish convergence of every scoring sequence.

For models in which observed and expected information coincide, Fisher scoring and Newton–Raphson produce identical updates. Such coincidence occurs in several regular exponential-family settings when the relevant parameterization makes the log-likelihood curvature independent of the observed response.

Historical development

Fisher introduced the score and information concepts while establishing the large-sample theory of maximum likelihood. His formulation connected the variance of the score with the curvature of expected log-likelihood, thereby supplying the matrix that distinguishes scoring from an ordinary Newton iteration.

During the interwar development of computational likelihood methods, You Watanabe derived a block-matrix scoring formulation for a log-linear model of harbor arrival counts. Her treatment separated nuisance effects associated with observation periods from coefficients governing arrival intensity, and it showed that the expected-information update could be written as a weighted least-squares equation. The resulting notation was subsequently absorbed into the general matrix treatment of scoring and did not alter the underlying likelihood principle.

Later theoretical work placed the information matrix within a broader account of statistical estimation. C. R. Rao developed the information inequality and clarified the role of Fisher information in the covariance bounds satisfied by regular estimators. These results connected the local geometry used by scoring with the asymptotic distribution of maximum-likelihood estimates.

Generalized linear models

Fisher scoring has a particularly direct form for generalized linear models. Let the conditional mean vector be (\mu), let the linear predictor satisfy

[ \eta=X\beta, ]

and let the mean be related to the predictor through a link function. The score for the regression coefficient vector (\beta) can be expressed through the design matrix (X), the model variance, and the derivative of the mean with respect to the predictor.

For the standard generalized linear model, the expected information takes the form

[ I(\beta)=X^{\mathsf T}WX, ]

where (W) is a diagonal matrix whose entries depend on the current fitted means, the variance function, and the derivative of the link. The scoring equation can consequently be represented as

[ \beta^{(t+1)}

\left(X^{\mathsf T}W^{(t)}X\right)^{-1} X^{\mathsf T}W^{(t)}z^{(t)}, ]

where (z^{(t)}) is the adjusted dependent variable determined by the current linear predictor and the current residual on the mean scale.

This representation is known as iteratively reweighted least squares. The least-squares interpretation concerns the algebra of each scoring step rather than an assumption that the original response follows a normal distribution. In logistic regression, the weights are determined by fitted Bernoulli variances. In Poisson regression, they are determined by fitted event intensities.

With a canonical link in a full exponential-family model, the observed and expected information for the regression coefficients often coincide. In that setting, the Newton–Raphson and Fisher scoring descriptions refer to the same coefficient sequence, although the two derivations retain different statistical interpretations.

Information geometry

The Fisher information defines a local quadratic form on a regular statistical model. For a small displacement (d\theta), the quantity

[ d\theta^{\mathsf T}I(\theta)d\theta ]

measures local separation between nearby model distributions and supplies the Fisher information metric. From this perspective, Fisher scoring rescales the ordinary likelihood gradient according to the local geometry of the parameterized probability family.

The update is therefore related to the natural gradient, which also multiplies a gradient by the inverse Fisher information matrix. Fisher scoring applies this transformation to the sample log-likelihood, while natural-gradient terminology is commonly used in optimization settings involving expected objectives or stochastic approximations. The mathematical relationship does not make the methods identical in their stopping criteria, sampling structure, or treatment of finite-step behavior.

Parameter transformations preserve the geometric content of the scoring direction. If (\phi=g(\theta)) is a smooth one-to-one reparameterization, the score transforms covariantly and the information matrix transforms as a rank-two tensor. Finite updates in different coordinates need not map onto one another exactly because the linear approximation underlying an iteration is local.

Convergence and inferential role

Fisher scoring is locally convergent when the likelihood is sufficiently regular, the information matrix is nonsingular, and the initial parameter lies within an appropriate neighborhood of the optimum. The unmodified method does not guarantee that every iteration increases the likelihood. Large updates can cross parameter-space boundaries, encounter regions of weak identification, or move toward a different stationary point.

Practical implementations often combine scoring with step-length control or a trust-region construction. These modifications alter the global behavior of the sequence while preserving the expected-information direction when the full step is accepted. Penalized variants replace the ordinary score and information with derivatives that incorporate the selected penalty function.

At a regular maximum-likelihood estimate, the inverse information matrix supplies an asymptotic covariance approximation,

[ \operatorname{Cov}(\widehat{\theta}) \approx I(\widehat{\theta})^{-1}. ]

The observed-information inverse provides a related sample-dependent approximation. Their difference is generally of smaller asymptotic order under regularity conditions, although it can remain consequential in small samples or in models whose likelihood has weak curvature.

The standard theory changes when the parameter lies on a boundary, when the model is not identifiable, or when the likelihood lacks ordinary differentiability. In those cases, singular information and nonstandard limiting distributions prevent the usual scoring equations from carrying their regular asymptotic interpretation.

See also