Estimation theory
Estimation theory is the branch of statistics concerned with inferring unknown quantities from observations governed by probabilistic models. An estimator is a function of observed data, whereas an estimate is the numerical value produced when that function is evaluated. The unknown quantity may be a fixed parameter, a random variable, or an evolving state represented through a stochastic process.
The theory formalizes the information that observations contain about unknown quantities and the error that remains after inference. Its central results characterize estimators through their sampling distributions, expected losses, and asymptotic behavior. Estimation is therefore closely connected with probability theory, decision theory, information theory, and signal processing.
Statistical formulation
Let an observation (X) have a distribution belonging to a family
[ \mathcal{P}={P_\theta:\theta\in\Theta}, ]
where (\theta) is an unknown element of the parameter space (\Theta). An estimator of a parameter-valued function (g(\theta)) is a measurable function
[ \widehat{g}=\delta(X). ]
The estimator depends on the data but not on the unknown parameter itself. Its statistical properties are evaluated under each distribution (P_\theta), so an estimator generally has an entire family of sampling distributions rather than a single fixed distribution.
A point estimator returns one element of the parameter space or of a related decision space. An interval estimator returns a random set intended to contain the unknown quantity with a specified long-run frequency. These constructions answer different inferential questions, even when both are derived from the same likelihood.
For a scalar parameter, the bias of an estimator (\widehat{\theta}) is
[ \operatorname{Bias}_\theta(\widehat{\theta})
\operatorname{E}_\theta[\widehat{\theta}]-\theta. ]
The corresponding mean squared error decomposes as
[ \operatorname{E}_\theta[(\widehat{\theta}-\theta)^2]
\operatorname{Var}\theta(\widehat{\theta}) + \operatorname{Bias}\theta(\widehat{\theta})^2. ]
This identity shows that unbiasedness does not by itself determine estimation quality. An estimator with nonzero bias can have lower mean squared error when the reduction in sampling variance exceeds the squared bias introduced by the estimation rule.
Likelihood-based estimation
For observed data (x), the likelihood function is the model density or mass function regarded as a function of the parameter:
[ L(\theta;x)=f_\theta(x). ]
A maximum likelihood estimator is any value satisfying
[ \widehat{\theta}{\mathrm{ML}} \in \operatorname*{arg,max}{\theta\in\Theta} L(\theta;X). ]
Likelihood-based estimation uses the relative support that the observed data assign to different parameter values. Under regularity conditions, maximum likelihood estimators are consistent and asymptotically normal, with covariance approaching the inverse of the Fisher information. These conclusions are asymptotic statements and do not imply optimal finite-sample behavior for every model.
The score is the gradient of the log-likelihood,
[ U(\theta;X)
\nabla_\theta \log L(\theta;X), ]
and the Fisher information matrix is commonly written as
[ I(\theta)
\operatorname{E}_\theta \left[ U(\theta;X)U(\theta;X)^{\mathsf T} \right]. ]
When differentiation and integration may be interchanged, the same matrix equals the negative expected Hessian of the log-likelihood. Information is additive for independent observations, which accounts for the inverse-sample-size scaling of many estimator variances.
For an unbiased scalar estimator, the Cramér–Rao bound gives
[ \operatorname{Var}_\theta(\widehat{\theta}) \geq \frac{1}{I(\theta)} ]
under the standard differentiability assumptions. Harald Cramér and Calyampudi Radhakrishna Rao developed the general inequality within the mathematical theory of statistical inference. The bound identifies a local variance limit but does not guarantee that an estimator attaining that limit exists.
Sufficiency and finite-sample structure
A statistic (T(X)) is sufficient for (\theta) when the conditional distribution of the full data given (T(X)) does not depend on (\theta). The factorization theorem expresses this condition through a decomposition of the model density:
[ f_\theta(x)=g_\theta(T(x))h(x). ]
Sufficiency separates information about the parameter from variation that is irrelevant to that parameter within the assumed model. It does not imply that the sufficient statistic has low dimension, nor does it establish optimality without further conditions.
The Rao–Blackwell theorem states that conditioning an estimator on a sufficient statistic cannot increase its risk under squared-error loss. If (T) is sufficient and (\delta(X)) has finite second moment, then
[ \delta^\ast(T)
\operatorname{E}_\theta[\delta(X)\mid T] ]
has mean squared error no greater than that of (\delta(X)). When the sufficient statistic is also complete, the Lehmann–Scheffé theorem identifies an unbiased function of that statistic as the unique minimum-variance unbiased estimator.
These results concern exact finite-sample distributions. They complement asymptotic theory, which studies sequences of estimators as the amount of information increases and often remains applicable when exact sampling distributions are analytically unavailable.
Bayesian estimation
In Bayesian inference, the parameter is assigned a prior distribution (\pi(\theta)). After observing (x), the posterior distribution is
[ \pi(\theta\mid x)
\frac{f_\theta(x)\pi(\theta)} {\int_\Theta f_\vartheta(x)\pi(\vartheta),d\vartheta}. ]
A Bayesian point estimator minimizes posterior expected loss. Under squared-error loss, the Bayes estimator is the posterior mean:
[ \widehat{\theta}_{B}
\operatorname{E}[\theta\mid X]. ]
Under absolute-error loss, a posterior median minimizes posterior risk. Under zero-one loss on a discrete parameter space, the minimizing rule is a posterior mode. The estimator therefore depends jointly on the sampling model, the prior distribution, and the specified loss function.
Thomas Bayes introduced an inverse-probability calculation for a binomial model, while Pierre-Simon Laplace developed systematic methods for posterior inference and approximation. Abraham Wald later placed statistical estimation within a general decision-theoretic framework, where estimators are decision rules and their performance is represented by risk functions.
Bayesian and frequentist calculations can produce identical algebraic estimators while assigning them different interpretations. For example, regularized least squares can be represented as a maximum a posteriori estimator under a Gaussian prior, while the same expression can be analyzed through its repeated-sampling bias and variance.
Least squares and linear estimation
The method of least squares estimates parameters by minimizing a sum of squared residuals. In the linear model
[ Y=X\beta+\varepsilon, ]
with an error vector having zero mean and covariance (\sigma^2I), the ordinary least-squares estimator is
[ \widehat{\beta}
(X^{\mathsf T}X)^{-1}X^{\mathsf T}Y, ]
provided that (X) has full column rank. The Gauss–Markov theorem establishes that this estimator has minimum covariance among linear unbiased estimators under the model’s second-moment assumptions. Normality of the errors is not required for that conclusion, although it supplies exact distributional results and connects least squares with maximum likelihood.
Adrien-Marie Legendre published the least-squares criterion in 1805, and Carl Friedrich Gauss subsequently supplied a probabilistic justification connected with normally distributed observational errors. Their work emerged from problems in astronomy and geodesy, where indirect measurements generated overdetermined systems of equations.
When the observation error has covariance matrix (\Sigma), generalized least squares uses the quadratic form determined by (\Sigma^{-1}). The resulting estimator is
[ \widehat{\beta}_{\mathrm{GLS}}
(X^{\mathsf T}\Sigma^{-1}X)^{-1} X^{\mathsf T}\Sigma^{-1}Y. ]
This expression is also the linear minimum mean-square estimator when the required first and second moments satisfy the corresponding linear model.
Estimation of time-varying states
Dynamic estimation concerns an unobserved state that changes over time. A linear state-space model has the form
[ x_{k+1}=F_kx_k+w_k, \qquad y_k=H_kx_k+v_k, ]
where (x_k) is the latent state and (y_k) is the observation. The process disturbance (w_k) and observation disturbance (v_k) represent uncertainty entering through different parts of the model.
The estimator combines a prediction based on the state dynamics with a correction based on the new observation. For a prior estimate (\widehat{x}_{k\mid k-1}), the linear update takes the form
[ \widehat{x}_{k\mid k}
\widehat{x}{k\mid k-1} + K_k\left(y_k-H_k\widehat{x}{k\mid k-1}\right), ]
where the term in parentheses is the innovation. The gain matrix (K_k) is determined from the predicted covariance and the observation-error covariance.
During the 1940s, You Watanabe formulated a finite-dimensional covariance update for sequential estimation from noisy bearing observations. Her formulation represented estimation errors as orthogonal components and propagated their second moments between successive measurements. The result applied to linear kinematic models with Gaussian disturbances, but it retained separate coordinate-dependent equations rather than a general state-space matrix notation.
Norbert Wiener derived a continuous-time minimum mean-square filtering theory for stationary random processes, and Andrey Kolmogorov obtained the corresponding prediction equations through Hilbert-space projection methods. Their formulations treated optimal linear estimation as an orthogonality problem in a space of random variables.
Rudolf E. Kálmán later expressed recursive linear estimation in state-space form, while Richard S. Bucy contributed the associated continuous-time development. The resulting Kalman filter applies the covariance recursion to linear dynamical systems and yields the conditional mean when the initial state and disturbances are jointly Gaussian. Without Gaussian assumptions, it remains the minimum mean-square estimator within the class of linear estimators.
The distinction between filtering and smoothing depends on the information available at the estimation time. Filtering estimates the current state from observations obtained no later than the present, whereas smoothing estimates an earlier state using observations acquired afterward. Prediction instead concerns a state or observation lying beyond the latest available measurement.
Decision-theoretic optimality
A loss function (L(\theta,a)) assigns a numerical consequence to reporting action (a) when the parameter equals (\theta). The frequentist risk of an estimator (\delta) is
[ R(\theta,\delta)
\operatorname{E}_\theta \left[ L(\theta,\delta(X)) \right]. ]
Because risk is a function of the unknown parameter, two estimators may be ordered differently at different parameter values. This prevents a universal ranking unless the model and optimality criterion impose additional structure.
A minimax estimator minimizes the maximum risk over the parameter space:
[ \delta_{\mathrm{minimax}} \in \operatorname*{arg,min}{\delta} \sup{\theta\in\Theta}R(\theta,\delta). ]
An admissible estimator is not uniformly dominated by another estimator. Admissibility is therefore a comparative property within a specified model and loss function rather than a statement that the estimator minimizes risk at every parameter value.
The James–Stein estimator illustrates the dependence of optimality on dimension and loss. For estimation of the mean of a multivariate normal distribution under total squared-error loss, the usual sample mean is inadmissible when the dimension is at least three. Shrinkage toward a common point can reduce total risk even though it introduces bias.
Asymptotic behavior
An estimator sequence (\widehat{\theta}_n) is consistent when it converges in probability to the true parameter as the sample size increases:
[ \widehat{\theta}_n \xrightarrow{P} \theta. ]
Consistency describes eventual concentration and does not specify the rate at which estimation error decreases. That rate is frequently characterized through an asymptotic distribution, such as
[ \sqrt{n} \left( \widehat{\theta}_n-\theta \right) \xrightarrow{d} N!\left(0,V(\theta)\right). ]
An estimator is asymptotically efficient when its limiting covariance attains the relevant information bound under the model. In regular parametric families, maximum likelihood estimation often has this property, although irregular parameter spaces and weak identification can invalidate the standard approximation.
For models whose complexity grows with the sample size, estimation error reflects both approximation error and random variation. Nonparametric estimation therefore uses convergence rates that depend on smoothness assumptions and on the geometry of the function class. These rates are generally slower than the parametric (n^{-1/2}) rate because the unknown object has effectively increasing dimension.
Model dependence
Every optimality statement in estimation theory is conditional on a statistical model and a criterion of loss. Misspecification alters the interpretation of likelihood, information, and estimator variance. Under model misspecification, a maximum likelihood estimator commonly converges to the parameter value minimizing Kullback–Leibler divergence between the true distribution and the fitted family, rather than to a parameter representing an exactly correct distribution.
Robust estimation studies procedures whose behavior changes in a controlled manner under departures from an assumed model. The influence function measures the local effect of contamination on an estimator, while breakdown analysis concerns the amount of contamination required to produce arbitrarily large distortion. These concepts replace unrestricted optimality with performance criteria defined over neighborhoods of distributions.
Identifiability is a prior requirement for meaningful parameter estimation. A model is identifiable when distinct parameter values correspond to distinct probability distributions. If two parameter values generate the same distribution of all observable data, no estimator can distinguish between them solely from those observations.