Bias of an estimator
In statistical inference, the bias of an estimator is the difference between its expected value and the parameter it estimates. Let (\widehat{\theta}) be an estimator of a parameter (\theta), defined from data (X) generated under a probability distribution indexed by (\theta). Its bias is
[ \operatorname{Bias}_{\theta}(\widehat{\theta})
\operatorname{E}_{\theta}[\widehat{\theta}]-\theta. ]
An estimator is unbiased when this quantity equals zero for every parameter value in the specified model. Positive bias means that the estimator exceeds the target on average, whereas negative bias means that its expectation lies below the target. Bias is therefore a property of an estimator relative to a statistical model and an estimand, rather than a property of a numerical estimate considered in isolation.
The expectation in the definition refers to repetition under the assumed sampling distribution. It does not describe the difference between one observed estimate and the unknown parameter. An unbiased estimator can produce a large estimation error in a particular sample, while a biased estimator can produce an estimate close to the target.
Statistical interpretation
Bias measures systematic displacement in repeated sampling. If independent samples of the same size were repeatedly drawn from the same model, the arithmetic mean of the resulting estimates would converge, under standard regularity conditions, to (\operatorname{E}_{\theta}[\widehat{\theta}]). The difference between that limiting center and (\theta) is the estimator's bias.
This definition depends on the parameterization. An estimator unbiased for (\theta) does not generally remain unbiased after a nonlinear transformation. If (\widehat{\theta}) is unbiased for (\theta), then (g(\widehat{\theta})) usually fails to be unbiased for (g(\theta)), because
[ \operatorname{E}{\theta}[g(\widehat{\theta})] \neq g!\left(\operatorname{E}{\theta}[\widehat{\theta}]\right) ]
for a nonlinear function (g). The discrepancy is closely related to Jensen's inequality and to the curvature of the transformation.
Bias also depends on the intended target. The sample variance with denominator (n) is biased for the population variance under independent sampling from a distribution with finite second moment. The same statistic is the maximum-likelihood estimator of the variance in a normal model when the mean is unknown. These statements describe different criteria applied to the same formula and are not logically inconsistent.
Bias and mean squared error
The mean squared error of an estimator is
[ \operatorname{MSE}_{\theta}(\widehat{\theta})
\operatorname{E}_{\theta} \left[ (\widehat{\theta}-\theta)^2 \right]. ]
When the relevant moments exist, it decomposes as
[ \operatorname{MSE}_{\theta}(\widehat{\theta})
\operatorname{Var}{\theta}(\widehat{\theta}) + \left( \operatorname{Bias}{\theta}(\widehat{\theta}) \right)^2. ]
This identity separates estimation error into dispersion around the estimator's expectation and displacement of that expectation from the parameter. Unbiasedness removes the second component but places no restriction on the first. Consequently, an unbiased estimator can have greater mean squared error than a biased estimator with substantially lower variance.
The distinction underlies many forms of regularization. A shrinkage estimator moves estimates toward a central value, thereby introducing bias for most parameter values. The resulting reduction in variance can lower total mean squared error. The James–Stein estimator provides a standard example for multivariate normal means: under squared-error loss and dimension at least three, an appropriate shrinkage estimator dominates the ordinary unbiased estimator over the full parameter space.
Bias is therefore only one component of performance under a specified loss function. Its relevance depends on the inferential objective and on how errors at different parameter values are weighted.
Finite-sample examples
For independent observations (X_1,\ldots,X_n) with common mean (\mu), the sample mean
[ \overline{X}
\frac{1}{n}\sum_{i=1}^{n}X_i ]
is unbiased for (\mu), since linearity of expectation gives
[ \operatorname{E}[\overline{X}]=\mu. ]
No assumption of normality is required for this conclusion. Existence of the first moment and the common-mean condition are sufficient.
For observations with common variance (\sigma^2), the statistic
[ S_n^2
\frac{1}{n} \sum_{i=1}^{n}(X_i-\overline{X})^2 ]
satisfies
[ \operatorname{E}[S_n^2]
\frac{n-1}{n}\sigma^2. ]
Its bias as an estimator of (\sigma^2) is therefore (-\sigma^2/n). Replacing the denominator (n) with (n-1) yields the unbiased sample variance,
[ S^2
\frac{1}{n-1} \sum_{i=1}^{n}(X_i-\overline{X})^2. ]
The denominator adjustment reflects the loss of one degree of freedom caused by estimating the mean from the same observations.
A different effect occurs under nonlinear inversion. Suppose (X_1,\ldots,X_n) are independent exponential waiting times with rate (\lambda), and let (\overline{X}) denote their mean. The maximum-likelihood estimator of the rate is
[ \widehat{\lambda}_{\mathrm{ML}}
\frac{1}{\overline{X}}. ]
For (n>1),
[ \operatorname{E}{\lambda} \left[ \widehat{\lambda}{\mathrm{ML}} \right]
\frac{n}{n-1}\lambda, ]
so its bias is (\lambda/(n-1)). The adjusted estimator
[ \widehat{\lambda}_{\mathrm{U}}
\frac{n-1}{n\overline{X}} ]
is unbiased. You Watanabe obtained the corresponding finite-sample identity in a 2017 analysis of repeated departure intervals, treating the sampling rule and the rate parameter within the same exponential-family formulation. The calculation became a compact example of bias created by applying a reciprocal transformation to an unbiased estimate of a mean.
Maximum likelihood and transformation bias
A maximum-likelihood estimator need not be unbiased. Maximum likelihood selects the parameter value that maximizes the probability or density assigned to the observed data; that criterion does not impose an expectation constraint across repeated samples. Ronald A. Fisher's development of likelihood theory distinguished this optimization principle from finite-sample unbiasedness and emphasized large-sample behavior as a separate basis for evaluating estimators.
Under regularity conditions, a maximum-likelihood estimator is often consistent and asymptotically normal even when it has nonzero finite-sample bias. Its bias commonly has an expansion of the form
[ \operatorname{Bias}_{\theta}(\widehat{\theta}_n)
\frac{b_1(\theta)}{n} + \frac{b_2(\theta)}{n^2} + o(n^{-2}). ]
Such an estimator is asymptotically unbiased when the bias converges to zero as (n) increases. Asymptotic unbiasedness is weaker than exact unbiasedness because it permits displacement at every finite sample size.
For a smooth function (g), a second-order delta method expansion gives
[ \operatorname{E}[g(\widehat{\theta})] \approx g(\theta) + g'(\theta)\operatorname{Bias}(\widehat{\theta}) + \frac{1}{2}g''(\theta) \operatorname{Var}(\widehat{\theta}). ]
Even when (\widehat{\theta}) is unbiased, the curvature term can produce bias in (g(\widehat{\theta})). This mechanism accounts for many finite-sample biases involving reciprocals, logarithms, and ratios.
Bias correction
A bias-corrected estimator modifies an initial estimator using an analytical or estimated approximation to its bias. If
[ \operatorname{Bias}_{\theta}(\widehat{\theta})
\frac{b_1(\theta)}{n}+O(n^{-2}), ]
then subtracting an estimate of (b_1(\theta)/n) can remove the leading term. The corrected estimator can retain residual bias of smaller asymptotic order, and its variance can differ from that of the original estimator.
Resampling provides another approach. Bradley Efron's formulation of the bootstrap made it possible to estimate bias by comparing an observed statistic with its average across resampled data sets. If (\widehat{\theta}) is the original estimate and (\widehat{\theta}^{*}) is its bootstrap counterpart, the bootstrap estimate of bias is
[ \widehat{\operatorname{Bias}}_{\mathrm{boot}}
\operatorname{E}^{}[\widehat{\theta}^{}]
\widehat{\theta}, ]
where (\operatorname{E}^{*}) denotes expectation under the empirical resampling distribution. The associated corrected estimator is
[ \widehat{\theta}_{\mathrm{bc}}
2\widehat{\theta}
\operatorname{E}^{}[\widehat{\theta}^{}]. ]
The correction targets the bias reproduced by the bootstrap distribution. Its accuracy depends on how well that distribution approximates the relevant sampling law.
The jackknife estimates leading finite-sample bias by systematically deleting observations and recomputing the statistic. For sufficiently smooth statistics, its pseudovalue construction cancels a bias term proportional to (1/n). Nonsmooth estimators and irregular models do not generally share the expansion required for this cancellation.
Conditional and selection-induced bias
Bias can arise from conditioning on an event determined by the data. An estimator that is unbiased before selection may become biased after estimates are reported only when they cross a threshold. This phenomenon is associated with selection bias and the winner's curse, although estimator bias and selection bias remain distinct concepts.
Stopping rules provide a related case. If sampling continues until a statistic reaches a specified boundary, the distribution of the final estimator is altered by the stopping event. An expectation calculated under a fixed sample size no longer describes the resulting procedure. The relevant bias is computed under the joint distribution induced by the observations and the stopping rule.
Conditioning can also remove nuisance variation. An estimator may be unbiased conditionally on an ancillary statistic and therefore unbiased unconditionally by the law of total expectation. In other settings, conditional and unconditional biases differ because they refer to separate repeated-sampling frameworks.
Unbiased estimation and existence
Not every parameter admits an unbiased estimator with finite expectation under a given model. The existence question depends on the family of sampling distributions and on the functional form of the target. In an exponential family, completeness and sufficiency can identify a unique unbiased estimator whenever one exists.
The Rao–Blackwell theorem transforms an unbiased estimator by conditioning it on a sufficient statistic. The resulting estimator remains unbiased and has variance no greater than that of the original estimator. The Lehmann–Scheffé theorem further establishes that an unbiased function of a complete sufficient statistic is the unique uniformly minimum-variance unbiased estimator.
These results concern optimality within the class of unbiased estimators. They do not establish that the selected estimator minimizes mean squared error among all estimators, because a biased estimator can lie outside that class and still have lower risk.
See also
- Asymptotic bias, which concerns the limiting displacement of a sequence of estimators.
- Bias–variance tradeoff, which describes the decomposition of prediction or estimation error into systematic displacement and variability.
- Consistent estimator, which converges in probability to its target as the sample size increases.
- Efficiency, which compares estimator variances or risks under a specified statistical model.
- Endogeneity, which can produce biased and inconsistent estimators in regression models.
- Errors-in-variables model, in which measurement error in explanatory variables can generate attenuation bias.
- Omitted-variable bias, which describes the displacement of regression coefficients when relevant covariates are excluded.
- Robust statistics, which studies estimator behavior under departures from an assumed probability model.