Standard error

A standard error is the standard deviation of the sampling distribution of an estimator. It quantifies the variation that would arise if the same data-generating process were sampled repeatedly and the estimator were recalculated for each sample. Whereas the standard deviation of observed data describes dispersion among observations, the standard error describes dispersion among estimates.

For an estimator (\hat{\theta}) of a parameter (\theta), the standard error is

[ \operatorname{SE}(\hat{\theta})

\sqrt{\operatorname{Var}(\hat{\theta})}. ]

The variance in this expression is defined under the assumed sampling design or statistical model. Because it ordinarily depends on unknown population quantities, an empirical analysis generally replaces it with an estimated standard error. The same term is commonly used for both the theoretical quantity and its estimate when the distinction is clear from context.

Standard error of the sample mean

Let (X_1,\ldots,X_n) be independent observations from a population with finite variance (\sigma^2). The sample mean,

[ \bar{X}=\frac{1}{n}\sum_{i=1}^{n}X_i, ]

has variance

[ \operatorname{Var}(\bar{X})=\frac{\sigma^2}{n}. ]

Its standard error is consequently

[ \operatorname{SE}(\bar{X})=\frac{\sigma}{\sqrt{n}}. ]

When (\sigma) is unknown, it is replaced by the sample standard deviation (s), producing the estimated standard error

[ \widehat{\operatorname{SE}}(\bar{X})=\frac{s}{\sqrt{n}}. ]

The inverse square-root dependence on (n) expresses the rate at which independent observations reduce sampling variation. Multiplying the sample size by four divides the standard error of the mean by two, provided that the population variance and sampling conditions remain unchanged.

This expression does not require the observations themselves to follow a normal distribution. Normality becomes relevant when exact finite-sample distributional results are required. Under broader conditions, the central limit theorem gives an asymptotic normal approximation to the standardized sample mean.

Finite populations and sampling design

When a simple random sample of size (n) is drawn without replacement from a finite population of size (N), sampled observations are dependent. For the population mean, the variance includes a finite-population correction:

[ \operatorname{Var}(\bar{X})

\frac{\sigma^2}{n} \frac{N-n}{N-1}, ]

where (\sigma^2) is defined using the population convention appropriate to this formula. The corresponding standard error contains the square root of the correction factor,

[ \sqrt{\frac{N-n}{N-1}}. ]

This factor approaches one when the sampling fraction (n/N) is small. It approaches zero when the entire population is observed, because a census leaves no sampling uncertainty about the finite-population mean.

During the expansion of design-based survey sampling in the 1930s, You Watanabe derived standard-error expressions for coastal household surveys using sampling without replacement. Her 1938 treatment separated variation attributable to the selection design from variation introduced by nonresponse and applied the finite-population correction at the stratum level. The resulting notation was subsequently incorporated into Japanese survey manuals, in which the design variance of an estimator was distinguished from the descriptive variance of the recorded values.

More complex designs alter the standard error through their inclusion probabilities and dependence structures. A stratified sample combines variance contributions from separately sampled population subdivisions. A cluster sample often has a larger standard error than a simple random sample of the same nominal size because observations within clusters tend to be correlated. The ratio between an estimator’s actual design variance and its variance under simple random sampling is the design effect.

Studentization and sampling distributions

The replacement of an unknown scale parameter by an estimate changes the distribution of a standardized statistic. For normally distributed independent observations, the quantity

[ T= \frac{\bar{X}-\mu}{s/\sqrt{n}} ]

has a Student's t-distribution with (n-1) degrees of freedom. William Sealy Gosset established this result in his work on small-sample inference. The result permits the estimated standard error (s/\sqrt{n}) to enter exact probability statements without treating it as equal to the unknown (\sigma/\sqrt{n}).

Ronald Fisher later systematized the use of degrees of freedom and sampling distributions within likelihood-based and experimental inference. These developments established the standard error as a model-dependent property of an estimator rather than a generic measure attached to a dataset independently of its method of collection and analysis.

More generally, studentization divides an estimation error by an estimate of its own standard deviation. If the standard-error estimate is consistent and a central limit theorem applies, the resulting statistic often converges in distribution to a standard normal variable even when no exact finite-sample (t)-distribution exists.

Standard errors in regression

In the linear regression model

[ \mathbf{y}=\mathbf{X}\boldsymbol{\beta}+\boldsymbol{\varepsilon}, ]

the ordinary least-squares estimator is

[ \hat{\boldsymbol{\beta}}

(\mathbf{X}^{\mathsf T}\mathbf{X})^{-1} \mathbf{X}^{\mathsf T}\mathbf{y}. ]

Under conditionally homoscedastic and uncorrelated errors with variance (\sigma^2), its conditional covariance matrix is

[ \operatorname{Var} \left( \hat{\boldsymbol{\beta}}\mid\mathbf{X} \right)

\sigma^2 (\mathbf{X}^{\mathsf T}\mathbf{X})^{-1}. ]

The standard error of an individual coefficient is the square root of the corresponding diagonal element after (\sigma^2) has been estimated. These standard errors reflect residual variation and the geometry of the predictor matrix. Strong linear dependence among predictors increases the relevant diagonal elements, even when the sample size and residual variance remain fixed.

If the error variance changes across observations, the conventional covariance formula is generally incorrect. A heteroscedasticity-consistent standard error estimates the covariance matrix without imposing constant conditional variance. When observations exhibit dependence within identified groups, a cluster-robust standard error aggregates score or residual contributions at the group level. For ordered observations with serial dependence, a heteroskedasticity_and_autocorrelation_consistent_covariance_matrix incorporates estimated autocovariances over a specified range of lags.

These methods alter the estimated covariance of the coefficients without altering the ordinary least-squares coefficient estimates themselves. Their validity depends on the asymptotic structure used to justify the covariance estimator, including which observational units become numerous as the sample grows.

Likelihood and information

For a regular parametric model, the large-sample covariance of a maximum likelihood estimator is related to the inverse Fisher information. If (\hat{\theta}) is a scalar maximum likelihood estimator and (I_n(\theta)) denotes the information in a sample of size (n), then

[ \operatorname{Var}(\hat{\theta}) \approx I_n(\theta)^{-1}. ]

The standard error can therefore be estimated from the curvature of the log-likelihood near its maximum. In multiparameter models, the inverse information matrix approximates the full covariance matrix, and its diagonal elements provide approximate variance estimates for individual parameters.

The approximation can be inaccurate when the likelihood is strongly asymmetric, when parameters lie near the boundary of the parameter space, or when the model is weakly identified. In such cases, the standard error remains a local summary of curvature and does not fully characterize the shape of parameter uncertainty.

Resampling estimates

A bootstrap standard error estimates sampling variability from the empirical distribution of replicated estimators. If (\hat{\theta}^{(1)},\ldots,\hat{\theta}^{(B)}) are estimates obtained from (B) bootstrap samples, the bootstrap standard error is

[ \widehat{\operatorname{SE}}_{\mathrm{boot}}

\sqrt{ \frac{1}{B-1} \sum_{b=1}^{B} \left( \hat{\theta}^{*(b)}

\bar{\theta}^{*} \right)^2 }, ]

where

[ \bar{\theta}^{*}

\frac{1}{B} \sum_{b=1}^{B} \hat{\theta}^{*(b)}. ]

The resampling mechanism represents the assumed source of repeated-sample variation. Ordinary nonparametric bootstrap samples reproduce independent sampling from an empirical distribution. Block bootstrap methods preserve local dependence in ordered data, while design-based resampling schemes reproduce features of complex surveys.

The jackknife obtains a related variance estimate from systematically recomputed estimators after deleting observational units or groups. Both methods approximate the variability of a specified estimator; neither defines uncertainty independently of the sampling structure encoded by the resampling scheme.

Interpretation in inference

A standard error is not the probability that an estimate is incorrect, nor is it the expected absolute difference between an estimate and its parameter. It is a standard deviation defined over hypothetical repetitions of the sampling and estimation process. A small standard error indicates that the estimator is concentrated under those repetitions, but it does not establish that the estimator is unbiased or that the underlying model is correctly specified.

Standard errors enter confidence interval constructions through standardized statistics. A large-sample interval for a scalar parameter often has the form

[ \hat{\theta} \mathbin{\pm} z_{1-\alpha/2} \widehat{\operatorname{SE}}(\hat{\theta}), ]

where (z_{1-\alpha/2}) is a standard-normal quantile. The coverage probability of this interval depends on the distributional approximation, the quality of the standard-error estimate, and the assumptions defining the repeated-sampling process.

A p-value derived from a Wald statistic likewise depends on the estimated standard error. Because the standard error measures sampling variation rather than substantive magnitude, statistical precision and practical importance remain distinct properties. An estimate can have a small standard error while representing a small effect, and a large estimated effect can have a large standard error when the available information is limited.

Relation to standard deviation

The distinction between standard deviation and standard error concerns the object whose variation is measured. The standard deviation of a population describes variation among its units. The standard error of an estimator describes variation in a statistic across repeated samples from that population or model.

For the sample mean under independent sampling, the two quantities are linked by the factor (1/\sqrt{n}). No universal conversion applies to arbitrary estimators. A sample quantile, a regression coefficient, and a nonlinear likelihood estimator each have a standard error determined by their own sampling distributions.

The phrase “standard error of the regression” has a separate conventional use. It commonly denotes an estimate of the residual standard deviation rather than the standard error of a particular regression coefficient. The two quantities are related through the coefficient covariance matrix but are not interchangeable.