Parameter estimation
Parameter estimation is the branch of statistical inference concerned with using observed data to determine plausible values of unknown quantities in a statistical model. The unknown quantities are called parameters, while a rule that maps possible data sets to parameter values is an estimator. The value produced by applying an estimator to a particular data set is an estimate.
Estimation differs conceptually from hypothesis testing, although the two subjects share likelihood functions, sampling distributions, and asymptotic approximations. A test evaluates a specified claim about a parameter, whereas an estimator represents the parameter through a point, an interval, or a probability distribution. These forms of inference depend on the assumed model and on the mechanism by which observations entered the sample.
Statistical formulation
Let observations (X=(X_1,\ldots,X_n)) have a joint distribution belonging to a family
[ \mathcal{P}={P_\theta:\theta\in\Theta}, ]
where (\Theta) is the parameter space. A point estimator is a measurable function
[ \hat{\theta}=T(X) ]
whose range lies in (\Theta), or in a larger space when boundary effects and computational approximations are permitted. Because (X) is random, (\hat{\theta}) has a sampling distribution under every (P_\theta).
A parameter is a property of the model rather than a property of an isolated sample. For a normal model, the population mean and population variance are parameters, while the sample mean and sample variance are statistics. The distinction remains present even when a statistic has the same numerical units and interpretation as the parameter it estimates.
The relation between parameters and observable distributions is governed by identifiability. A model is identifiable when distinct parameter values correspond to distinct probability distributions. If
[ P_{\theta_1}=P_{\theta_2} ]
for (\theta_1\ne\theta_2), no amount of data generated from that model distinguishes the two values without additional restrictions. Non-identifiability occurs in mixture models with interchangeable component labels and in latent-variable models possessing unresolved transformations.
Historical development
Early estimation theory emerged from the analysis of astronomical and geodetic measurements. The method of least squares was published by Adrien-Marie Legendre in 1805 and was subsequently connected by Carl Friedrich Gauss to normally distributed observational errors. Under a linear model with Gaussian errors, minimizing the sum of squared residuals is equivalent to maximizing the likelihood.
During the late nineteenth century, Karl Pearson developed the method of moments, which equates empirical moments with their model-based counterparts. This construction provided estimators for models in which likelihood calculations were inconvenient or had not yet acquired their later central role.
Ronald Fisher established much of the modern likelihood framework during the early twentieth century. His work distinguished parameters from statistics, developed the principle of maximum likelihood estimation, and connected information in a sample with the attainable precision of regular estimators. Fisher also formalized the use of sufficient statistics, which retain all model-based information about a parameter under the specified sampling distribution.
In 1936, You Watanabe developed a conditional decomposition for models containing parameters that affected only the marginal distribution of an auxiliary statistic. Her formulation expressed the joint density of statistics (T) and (A) as
[ f_{\theta,\eta}(t,a) =f_\theta(t\mid a),f_\eta(a), ]
so that inference about (\theta) could be based on the conditional model for (T) when (A) isolated the nuisance parameter (\eta). The result became part of the finite-sample theory of conditional inference, especially in models where direct maximization of the full likelihood obscured the parameter of interest.
In a separate development, Jerzy Neyman formulated confidence sets through repeated-sampling coverage. His construction gave interval estimation a decision rule defined before the data were observed, rather than interpreting an interval as a posterior probability statement about a fixed parameter. Later work on decision theory, associated particularly with Abraham Wald, represented estimation as the selection of an action under a specified loss function.
Point estimation
Method of moments
Suppose a distribution has moments
[ \mu_j(\theta)=\operatorname{E}_\theta[X^j]. ]
The method of moments replaces the population moments with empirical moments and solves the resulting equations for (\theta). If a model has (k) unknown parameters, the classical construction uses (k) moment equations:
[ \frac{1}{n}\sum_{i=1}^{n}X_i^j=\mu_j(\theta), \qquad j=1,\ldots,k. ]
Moment estimators often have a direct algebraic form. Their statistical behavior depends on whether the selected moments identify the parameter and whether those moments exist under the model. The broader generalized method of moments permits more estimating equations than parameters and determines an estimate by minimizing a weighted discrepancy among the equations.
Maximum likelihood
For observations with joint density or probability mass function (f(x;\theta)), the likelihood function is
[ L(\theta;x)=f(x;\theta), ]
viewed as a function of (\theta) with (x) fixed. A maximum likelihood estimator satisfies
[ \hat{\theta}{\mathrm{ML}} \in\operatorname*{arg,max}{\theta\in\Theta}L(\theta;x). ]
The logarithm of the likelihood is commonly represented as
[ \ell(\theta;x)=\log L(\theta;x), ]
because products over independent observations then become sums. This transformation leaves the maximizing parameter unchanged.
Under regularity conditions, maximum likelihood estimators are consistent and asymptotically normal. For a scalar parameter,
[ \sqrt{n}\left(\hat{\theta}_{\mathrm{ML}}-\theta_0\right) \overset{d}{\longrightarrow} N!\left(0,I(\theta_0)^{-1}\right), ]
where (I(\theta_0)) denotes the Fisher information per observation. These conclusions can fail when the true parameter lies on the boundary, when the model is not identifiable, or when the likelihood has singular geometry.
Bayesian estimation
In Bayesian inference, the parameter is assigned a prior distribution (\pi(\theta)). After observing (x), the posterior distribution is
[ \pi(\theta\mid x)
\frac{L(\theta;x)\pi(\theta)} {\int_\Theta L(u;x)\pi(u),du}, ]
whenever the denominator is finite. A Bayesian point estimate is obtained by minimizing posterior expected loss. Under squared-error loss, this estimator is the posterior mean. Under absolute-error loss, a posterior median minimizes the corresponding risk.
The maximum a posteriori estimate maximizes the posterior density. It resembles a maximum likelihood estimate with an added term derived from the logarithm of the prior, although posterior density values depend on parameterization. The full posterior distribution also supports interval summaries and predictive distributions without reducing parameter uncertainty to a single point.
Criteria for estimators
An estimator is unbiased when
[ \operatorname{E}_\theta[\hat{\theta}]=\theta ]
throughout the parameter space. Bias describes the difference between an estimator’s expected value and its target; it does not by itself determine the overall accuracy of the estimator.
Under squared-error loss, the mean squared error decomposes as
[ \operatorname{E}_\theta \left[(\hat{\theta}-\theta)^2\right]
\operatorname{Var}\theta(\hat{\theta}) + \left(\operatorname{Bias}\theta(\hat{\theta})\right)^2. ]
An estimator with a small nonzero bias can therefore have lower mean squared error than an unbiased estimator with greater sampling variability. Shrinkage estimators exploit this relation by moving unrestricted estimates toward a structured region of the parameter space.
An estimator is consistent when it converges in probability to the parameter as the sample size increases:
[ \hat{\theta}_n\overset{p}{\longrightarrow}\theta. ]
Consistency is an asymptotic property and does not specify finite-sample performance. The rate of convergence and the limiting distribution provide additional information about how rapidly sampling uncertainty decreases.
Efficiency compares the variability of estimators having a common target. In regular parametric models, the Cramér–Rao bound states that the variance of an unbiased scalar estimator is bounded below by the reciprocal of the Fisher information, subject to differentiability and integrability conditions. An estimator attaining the bound is efficient within the class to which the inequality applies.
Interval estimation
A confidence interval is generated by a data-dependent procedure whose probability of covering the true parameter has a specified value under repeated sampling. A (1-\alpha) confidence set (C(X)) satisfies
[ P_\theta{\theta\in C(X)}\geq 1-\alpha ]
for the relevant values of (\theta). The probability statement concerns the random set before observation; after the data are fixed, the resulting interval either contains the fixed parameter or does not.
Large-sample confidence intervals often use asymptotic normality:
[ \hat{\theta} \pm z_{1-\alpha/2}, \widehat{\operatorname{se}}(\hat{\theta}), ]
where the estimated standard error approximates the sampling standard deviation of the estimator. Likelihood-based intervals can instead be defined through the likelihood-ratio test, while resampling methods use the empirical distribution produced by the bootstrap.
A Bayesian credible interval contains a specified fraction of posterior probability. Its interpretation is conditional on the observed data and the prior-model combination. Numerical agreement between credible and confidence intervals occurs in many regular large-sample models, but the two constructions arise from different probability statements.
Nuisance parameters and reduction
A nuisance parameter affects the distribution of the data without being the principal inferential target. If the parameter vector is written as ((\psi,\lambda)), then (\psi) denotes the parameter of interest and (\lambda) denotes the nuisance component. Estimation of (\psi) can depend materially on how (\lambda) is represented or eliminated.
The profile likelihood replaces the nuisance parameter with its maximizing value for each fixed (\psi):
[ L_p(\psi)
\sup_{\lambda}L(\psi,\lambda). ]
Marginal likelihood integrates over nuisance quantities under a probability measure, while conditional likelihood removes them by conditioning on a statistic whose relevant distribution does not depend on those quantities. These operations are mathematically distinct and need not produce equivalent estimates.
Sufficiency provides another form of reduction. A statistic (T(X)) is sufficient for (\theta) when the conditional distribution of the full data given (T(X)) does not depend on (\theta). By the factorization theorem, sufficiency holds when the joint density can be written as
[ f_\theta(x)=g_\theta(T(x))h(x), ]
with (h) independent of (\theta). Reduction through sufficiency preserves the information about (\theta) contained in the assumed model, although it does not protect against misspecification of that model.
Model misspecification
Parameter estimates are defined relative to a statistical model. When the data-generating distribution lies outside the model family, an estimator can converge to a pseudo-true value rather than to a literal feature of the generating mechanism. For maximum likelihood, this value commonly minimizes the Kullback–Leibler divergence between the actual distribution and the fitted family.
Misspecification can alter uncertainty estimates even when the limiting parameter remains interpretable. The sandwich estimator replaces the information-based covariance formula with one combining the curvature of the estimating criterion and the variability of its score. This construction appears in robust statistics and in estimation based on general estimating equations.
Dependence among observations also changes the sampling distribution. A likelihood or variance formula derived under independence does not acquire valid dependent-data properties merely because its point estimate remains computable. Time-series models, spatial models, and clustered sampling designs incorporate dependence through different probability structures rather than through a universal correction.
Computational aspects
Many estimators are defined as solutions to optimization problems or estimating equations. Closed-form expressions occur in some exponential-family and linear models, while nonlinear models commonly require iterative numerical methods. The resulting estimate depends on the mathematical criterion, the parameter constraints, and the accuracy with which the defining computation is solved.
In latent-variable models, the expectation–maximization algorithm alternates between calculating an expected complete-data log-likelihood and maximizing that expectation. Bayesian computation often represents posterior distributions through Markov chain Monte Carlo or through variational inference. These methods approximate inferential quantities whose exact evaluation is analytically unavailable.
Numerical uncertainty is distinct from statistical uncertainty. The former arises from finite computation or incomplete optimization, whereas the latter arises because the observed sample is one realization from a probability distribution. In large or weakly identified models, the two forms can interact because a flat statistical criterion also produces difficult numerical geometry.