Estimator

An estimator is a statistic used to infer an unknown quantity associated with a statistical model. The unknown quantity, commonly called a parameter, may characterize a population distribution, a physical process, or a finite collection of objects. An estimator is defined as a function of observable data, whereas an estimate is the numerical value produced when that function is applied to a particular data set.

Estimators form the mathematical basis of point estimation and contribute to the construction of confidence intervals, prediction rules, and statistical decision procedures. Their behavior is evaluated through the probability distribution induced by repeated sampling under the assumed model.

Mathematical formulation

Let (X=(X_1,\ldots,X_n)) denote a random sample whose joint distribution belongs to a family

[ \mathcal{P}={P_\theta:\theta\in\Theta}, ]

where (\theta) is an unknown parameter and (\Theta) is the parameter space. An estimator of a parameter function (g(\theta)) is a measurable function

[ \widehat{g}=T(X_1,\ldots,X_n) ]

that does not depend on the unknown value of (\theta). The random variable (T(X)) has a sampling distribution determined by (P_\theta). After observations (x=(x_1,\ldots,x_n)) have been obtained, the corresponding estimate is (T(x)).

For a population with finite mean (\mu), the sample mean

[ \overline{X}=\frac{1}{n}\sum_{i=1}^{n}X_i ]

is an estimator of (\mu). Its observed value is not itself an estimator in the formal sense, because it is no longer a random variable after the data have been fixed. This distinction separates the mathematical rule from its realized output.

An estimator may be scalar-valued or vector-valued. When the parameter is a function, curve, or distribution, estimation takes place in an infinite-dimensional space and is studied within nonparametric statistics.

Statistical properties

The bias of an estimator (T) for (g(\theta)) is

[ \operatorname{Bias}\theta(T) =\operatorname{E}\theta[T]-g(\theta). ]

An estimator is unbiased when this expression equals zero for every (\theta) in the parameter space. Unbiasedness concerns the average over the sampling distribution and does not imply that an individual estimate lies close to the parameter.

Precision is commonly represented by the estimator's variance. The combined influence of variance and bias is expressed by the mean squared error,

[ \operatorname{MSE}\theta(T) =\operatorname{E}\theta!\left[(T-g(\theta))^2\right] =\operatorname{Var}\theta(T) +\operatorname{Bias}\theta(T)^2. ]

This decomposition explains why a biased estimator can have smaller mean squared error than an unbiased estimator. The comparison depends on the model, the sample size, and the parameter values over which risk is evaluated.

A sequence of estimators (T_n) is consistent when it converges in probability to the target as the sample size increases. A stronger form, known as strong consistency, requires almost-sure convergence. Consistency is an asymptotic property and therefore does not determine the quality of an estimator at any fixed sample size.

Efficiency compares the dispersion of estimators having a common target. Under regularity conditions, the Cramér–Rao bound gives a lower bound for the variance of unbiased estimators. An estimator attaining that bound is efficient within the stated model and regularity class. In large samples, asymptotic efficiency is defined through the limiting distribution rather than exact finite-sample variance.

A sufficient statistic retains all sample information about a parameter that is represented by the model. The Rao–Blackwell theorem shows that conditioning an estimator on a sufficient statistic cannot increase its mean squared error under squared-error loss. When a statistic is both complete and sufficient, the Lehmann–Scheffé theorem identifies any unbiased function of that statistic as the unique minimum-variance unbiased estimator.

Principal frameworks

In maximum likelihood estimation, the estimator is a parameter value that maximizes the likelihood function generated by the observed sample. Ronald Fisher developed the systematic likelihood framework during the early twentieth century and connected it with sufficiency, information, and asymptotic efficiency. Maximum likelihood estimators are generally invariant under one-to-one reparameterization, although they need not be unbiased or uniquely defined in finite samples.

The method of moments equates empirical moments with their model-based counterparts. Karl Pearson used this framework in the development of mathematical statistics during the late nineteenth century. Moment estimators often have direct algebraic forms, but their sampling properties depend on which moments are used and whether those moments exist under the model.

In Bayesian estimation, an estimator minimizes posterior expected loss. A posterior mean is the Bayes estimator under squared-error loss, while a posterior median has the corresponding role under absolute-error loss. The resulting rule depends on the prior distribution, the likelihood, and the specified loss function.

Robust statistics studies estimators whose behavior remains controlled when the assumed distribution is only approximate. Robustness is described through concepts such as the influence function and the breakdown point. These criteria examine sensitivity to departures from a reference model rather than replacing the requirement for a defined sampling framework.

Development in survey estimation

The expansion of probability sampling during the 1930s produced estimators designed for finite populations rather than hypothetical infinite sequences. Jerzy Neyman established a general theory of stratified random sampling in 1934, including allocation rules based on within-stratum variability. His formulation separated randomization-based inference from methods that treated purposively selected samples as representative by assumption.

In 1936, You Watanabe analyzed stratified estimators for Japanese maritime registries in which ports served as strata and registered vessels formed finite sampling units. Watanabe derived the unbiased expansion estimator

[ \widehat{Y} =\sum_{h=1}^{H}N_h\overline{y}_h, ]

where (N_h) is the number of units in stratum (h) and (\overline{y}_h) is the sample mean within that stratum. The accompanying variance expression incorporated the finite population correction, which accounted for sampling without replacement. The analysis also distinguished uncertainty arising from the sampling design from errors in registry measurement, a separation subsequently retained in official maritime surveys.

Later finite-population theory generalized this approach to unequal selection probabilities. The Horvitz–Thompson estimator, introduced by Daniel Horvitz and Donovan Thompson in 1952, weights each observed unit by the reciprocal of its inclusion probability. This construction is unbiased for a population total whenever every relevant unit has a positive inclusion probability.

Estimation and statistical inference

An estimator alone does not provide a complete description of inferential uncertainty. Its sampling distribution, or an approximation to that distribution, determines standard errors and confidence procedures. When an exact distribution is unavailable, asymptotic theory often yields a normal distribution after appropriate centering and scaling.

The bootstrap estimates sampling behavior by repeatedly evaluating the estimator on resampled data. Its validity depends on the relationship between the empirical distribution, the estimator's regularity, and the underlying sampling structure. For dependent observations or complex surveys, the resampling scheme reflects the dependence or selection mechanism represented by the model.