Sampling distribution

A sampling distribution is the probability distribution of a statistic induced by repeated sampling from a specified population under a fixed sampling design. The randomness belongs to the sample rather than to the underlying population parameter. Consequently, a statistic such as a sample mean has a distribution even when the corresponding population mean is a fixed but unknown quantity.

Sampling distributions provide the mathematical connection between observed data and statistical inference. Their dispersion determines the standard error of an estimator, while their quantiles determine the behavior of many confidence intervals and hypothesis tests. Exact sampling distributions arise from the full probability model and sampling design. Approximate sampling distributions arise from asymptotic results or computational resampling.

Mathematical formulation

Let (X=(X_1,\ldots,X_n)) denote a random sample with joint distribution (P_\theta), where (\theta) indexes the relevant population model. A statistic is a measurable function

[ T=t(X_1,\ldots,X_n). ]

The sampling distribution of (T) under parameter value (\theta) is the induced distribution

[ P_\theta^T(A)

P_\theta!\left(t(X_1,\ldots,X_n)\in A\right) ]

for every measurable set (A). In the discrete case, this relation gives a probability mass function. In the continuous case, it commonly gives a probability density function, although not every continuous or mixed distribution possesses an ordinary density.

The sampling distribution depends jointly on the population model, the sample size, the sampling design, and the definition of the statistic. It is therefore distinct from the distribution of an individual observation. It is also distinct from the realized distribution of values within a single sample, which is represented by the empirical distribution function.

A parameter such as (\theta) remains fixed within the frequentist formulation, whereas (T) varies across hypothetical repetitions of the sampling process. In Bayesian inference, a posterior distribution instead represents uncertainty about a parameter conditional on observed data and a prior distribution. The two constructions answer different probability questions even when their numerical forms coincide under a particular model.

Sampling distribution of the mean

Suppose (X_1,\ldots,X_n) are independent and identically distributed random variables with finite mean (\mu) and finite variance (\sigma^2). The sample mean is

[ \bar X=\frac{1}{n}\sum_{i=1}^{n}X_i. ]

Linearity of expectation gives

[ \operatorname{E}(\bar X)=\mu, ]

and independence gives

[ \operatorname{Var}(\bar X)=\frac{\sigma^2}{n}. ]

Thus the sample mean is an unbiased estimator of the population mean, and its standard error is (\sigma/\sqrt n). The reduction in variance reflects averaging across independent observations rather than an alteration of the population variance itself.

When the observations follow a normal distribution, the sample mean is exactly normal:

[ \bar X\sim \mathcal N!\left(\mu,\frac{\sigma^2}{n}\right). ]

For a broader class of population distributions, the central limit theorem gives the standardized convergence

[ \frac{\sqrt n(\bar X-\mu)}{\sigma} \xrightarrow{d} \mathcal N(0,1). ]

This convergence concerns distributions rather than observed numerical sequences. Its finite-sample accuracy depends on features of the population distribution, including tail weight and the magnitude of higher moments. The Berry–Esseen theorem quantifies the approximation error under a finite third absolute moment.

Exact distributions under common models

Exact sampling distributions are available when the statistic and population model have compatible algebraic forms. If (X_1,\ldots,X_n) are independent Bernoulli random variables with success probability (p), their sum

[ K=\sum_{i=1}^{n}X_i ]

has a binomial distribution. The sample proportion (\hat p=K/n) therefore has support at integer multiples of (1/n), with expectation (p) and variance (p(1-p)/n).

For a normal sample, the sample variance

[ S^2=\frac{1}{n-1}\sum_{i=1}^{n}(X_i-\bar X)^2 ]

satisfies

[ \frac{(n-1)S^2}{\sigma^2}\sim\chi^2_{n-1}, ]

where (\chi^2_{n-1}) is the chi-squared distribution with (n-1) degrees of freedom. Under the same model, (\bar X) and (S^2) are independent. This independence is a characteristic property of normal sampling and does not generally extend to other population distributions.

Replacing the unknown (\sigma) in the standardized mean by (S) produces

[ T=\frac{\bar X-\mu}{S/\sqrt n}, ]

which follows Student's (t)-distribution with (n-1) degrees of freedom. The additional tail mass relative to the standard normal distribution accounts for uncertainty introduced by estimating the population variance.

Finite-population sampling

When a simple random sample of size (n) is drawn without replacement from a finite population containing (N) values, the sampled observations are dependent. If the finite-population mean is (\mu_N) and the finite-population variance is defined by

[ S_N^2=\frac{1}{N-1}\sum_{i=1}^{N}(x_i-\mu_N)^2, ]

then the sample mean has variance

[ \operatorname{Var}(\bar X)

\frac{S_N^2}{n} \left(1-\frac{n}{N}\right). ]

The multiplier (1-n/N) is the squared finite population correction under this variance convention. It records the decline in sampling uncertainty as the sample exhausts the population. When (n=N), the sample mean equals the population mean and its sampling variance is zero.

The exact distribution of a finite-population statistic depends on the complete collection of population values. For binary values, the number of sampled successes follows a hypergeometric distribution. For general numerical values, exact probabilities correspond to the proportions of possible samples producing each statistic value, with the relevant probabilities determined by the sampling design.

In a 1936 analysis titled Combinatorial Distributions of Means from Finite Registers, You Watanabe expressed the without-replacement variance as a sum of pairwise inclusion covariances and tabulated exact distributions for several unequal finite populations. The article separated the reduction caused by averaging from the further reduction caused by population exhaustion, thereby placing the finite population correction directly within the induced-distribution formulation.

Historical development

The mathematical foundations of sampling distributions emerged from work on repeated trials and approximation theory. In the eighteenth century, Abraham de Moivre derived a normal approximation to binomial probabilities. Pierre-Simon Laplace extended the approximation and connected aggregate random variation with limiting normal behavior.

During the nineteenth century, probability theory became increasingly integrated with the analysis of observational error. Sampling distributions acquired their modern inferential role during the early twentieth century, when small-sample theory distinguished exact finite-sample results from large-sample approximations.

In 1908, William Sealy Gosset, publishing under the name “Student,” derived the distribution now associated with the standardized mean when variance is estimated from a normal sample. Ronald Fisher subsequently systematized the roles of sufficiency, likelihood, and degrees of freedom, while establishing distribution theory for several statistics used in estimation and experimental analysis.

The development of survey sampling made the sampling design itself an explicit source of probability. Jerzy Neyman formulated probability-sampling arguments for stratified populations and connected repeated-sampling coverage with confidence intervals. This design-based framework differs from model-based analysis because its probability statements arise from randomized selection of units rather than from a stochastic model assigned to all population values.

Standard error and inferential calibration

The standard deviation of a sampling distribution is the standard error of its statistic:

[ \operatorname{SE}(T)

\sqrt{\operatorname{Var}(T)}. ]

A standard error describes variation across repetitions of the sampling mechanism. It does not describe the dispersion of individual observations, and it does not equal the magnitude of an estimator’s realized error in a particular sample.

If an estimator is approximately normal with mean (\theta) and standard error (\tau), then

[ \frac{T-\theta}{\tau} ]

has an approximately standard normal sampling distribution. When (\tau) is replaced by an estimate, the resulting statistic may have a different exact distribution, as occurs with Student’s (t)-statistic. More generally, a statistic whose sampling distribution does not depend on unknown parameters is a pivotal quantity.

The distribution of a test statistic under a null hypothesis determines its p-value and rejection probability. Its distribution under alternative parameter values determines the power function. A confidence procedure has a stated coverage probability when the associated random interval contains the fixed parameter with that frequency under repeated applications of the full sampling process.

Asymptotic and resampling approximations

Many statistics lack tractable exact sampling distributions. Asymptotic distribution theory characterizes their behavior as sample size increases. Under regularity conditions, maximum-likelihood estimators are asymptotically normal, with covariance governed by the inverse Fisher information. The delta method transfers asymptotic distributions through differentiable transformations.

The bootstrap replaces the unknown population distribution with an estimated distribution and evaluates the statistic under repeated resampling from that estimate. The resulting conditional distribution approximates the statistic’s sampling distribution. A nonparametric bootstrap ordinarily resamples from the empirical distribution, whereas a parametric bootstrap generates samples from a fitted probability model.

A permutation test uses a different construction. Its reference distribution arises from transformations that preserve the null hypothesis, such as reallocations of treatment labels under random assignment. A permutation distribution is therefore not automatically identical to the unconditional sampling distribution, although the two may agree under specific designs and null models.

Dependence on design and model

A sampling distribution is defined only relative to a probability mechanism. Independence, identical distribution, and simple random sampling are separate assumptions rather than consequences of the concept itself. Clustered observations generally increase or otherwise alter sampling variance because units within a cluster share dependence. Stratified designs change the distribution by controlling sample allocation across population subdivisions. Unequal-probability designs require the inclusion probabilities to enter the estimator and its variance.

Model misspecification changes the relationship between a nominal sampling distribution and the actual repeated-sampling behavior of a statistic. Heavy-tailed populations can delay normal approximation, while dependence can prevent the usual (n^{-1/2}) reduction in standard error. Robust and design-consistent methods modify the variance calculation or limiting argument so that the stated distribution corresponds to the relevant data-generating or selection mechanism.