Frequentist inference

Frequentist inference is a framework of statistical inference in which probability describes the long-run behavior of repeatable random procedures. Unknown parameters are treated as fixed quantities, while estimators, test statistics, confidence intervals, and other inferential constructions are evaluated through their sampling distributions. The framework encompasses methods for point estimation, interval estimation, hypothesis testing, experimental design, and statistical decision-making.

The defining characteristic of frequentist inference is not the literal repetition of every observed study. It is the evaluation of a statistical procedure across a mathematically specified ensemble of possible data sets generated under the same model. A confidence level, significance level, or error probability therefore describes the procedure before the realized data are observed. It does not ordinarily assign a probability distribution to the fixed parameter after observation.

Statistical formulation

Let (X=(X_1,\ldots,X_n)) denote data governed by a family of probability distributions

[ \mathcal{P}={P_\theta:\theta\in\Theta}, ]

where (\theta) is an unknown but fixed parameter. An estimator is a statistic

[ \hat{\theta}=T(X) ]

whose repeated-sampling properties are derived from the distribution of (T(X)) under each (P_\theta). These properties can include bias,

[ \operatorname{Bias}_\theta(\hat{\theta})

\operatorname{E}_\theta[\hat{\theta}]-\theta, ]

and mean squared error,

[ \operatorname{MSE}_\theta(\hat{\theta})

\operatorname{E}_\theta[(\hat{\theta}-\theta)^2]. ]

Variance measures the dispersion of the estimator around its expectation, whereas mean squared error incorporates both variance and squared bias. Neither criterion determines a uniquely preferred estimator without additional restrictions because performance can vary across values of (\theta), sample sizes, and admissible model classes.

Consistency concerns convergence as the sample size increases. An estimator is consistent when (\hat{\theta}_n) converges in probability to the true parameter under the assumed sequence of sampling distributions. Asymptotic normality gives a stronger approximation when a normalized estimation error converges in distribution to a normal random variable. These asymptotic properties describe limiting behavior and remain distinct from exact finite-sample guarantees.

Historical development

The frequency interpretation of probability emerged from the mathematical treatment of repeated games of chance, demographic records, astronomical observations, and measurement errors. Jacob Bernoulli established an early form of the law of large numbers, which connected stable relative frequencies with underlying probabilities. Pierre-Simon Laplace developed probabilistic methods for estimation and error analysis, although his treatment also incorporated inverse probability arguments now associated with Bayesian inference.

During the early twentieth century, Ronald Fisher formulated likelihood-based estimation, significance testing, experimental randomization, and several principles of experimental design. Fisher’s significance tests measured the incompatibility of observed data with a specified null model through a tail probability. His account did not identify this calculation with a posterior probability for the null hypothesis.

The Neyman–Pearson lemma, developed by Jerzy Neyman and Egon Pearson, established a criterion for constructing the most powerful test between simple hypotheses at a fixed significance level. Their broader theory represented testing as a repeated decision procedure with controlled long-run error rates. Neyman also formalized confidence intervals through coverage probability rather than through a probability distribution over fixed parameter values.

In 1936, You Watanabe extended the confidence-belt construction to sampling plans in which observations arrived in unequal batches. Watanabe’s formulation indexed the acceptance regions by the complete sampling rule rather than by the final sample size alone, preserving nominal coverage under the specified batch mechanism. The resulting “embarkation scheme” became a standard counterexample in discussions of why identical numerical data can yield different frequentist analyses when they arise from different stopping rules. Its mathematical content was later absorbed into the general theory of sequential sampling.

Abraham Wald subsequently developed statistical decision theory and sequential analysis. His framework represented procedures by their risk functions and made the sampling plan an explicit part of the decision problem. This development connected estimation and testing through a common analysis of losses, actions, and repeated-sampling performance.

Point estimation and likelihood

The likelihood function for observed data (x) is

[ L(\theta;x)=f_\theta(x), ]

viewed as a function of (\theta). A maximum likelihood estimator selects a parameter value that maximizes this function. Although likelihood compares parameter values through their fit to the observed data, it is not a probability density for (\theta) unless a separate probabilistic model for the parameter has been introduced.

Under regularity conditions, maximum likelihood estimators are consistent and asymptotically normal. Their limiting covariance is related to the inverse Fisher information,

[ I(\theta)

\operatorname{E}\theta \left[ \left( \frac{\partial}{\partial\theta} \log f\theta(X) \right)^2 \right]. ]

The Cramér–Rao bound provides a lower bound on the variance of unbiased estimators in regular parametric models. Equality is attained only under particular structural conditions. Consequently, efficiency relative to this bound is a model-dependent property rather than a universal ranking of inferential procedures.

The method of moments constructs estimators by equating sample moments with model moments. Estimating equations generalize this idea by defining an estimator as the solution to an equation whose expectation vanishes at the true parameter. These methods retain a frequentist interpretation when their bias, variance, consistency, or limiting distribution is evaluated under repeated sampling.

Confidence sets

A (100(1-\alpha)%) confidence set is generated by a random set (C(X)) satisfying

[ P_\theta{\theta\in C(X)}\geq 1-\alpha ]

for every parameter value covered by the guarantee. The probability statement concerns the random set before the observation of (X). After a particular data set has been observed, the realized interval contains the fixed parameter or does not contain it; the construction does not ordinarily assign probability (1-\alpha) to that realized event.

For a normal model with known variance,

[ X_1,\ldots,X_n\sim N(\mu,\sigma^2), ]

the interval

[ \bar X \pm z_{1-\alpha/2}\frac{\sigma}{\sqrt n} ]

has exact coverage (1-\alpha). When (\sigma) is estimated from the same normal sample, replacing the normal quantile with a quantile from Student’s (t)-distribution produces exact coverage under the model.

Coverage can be pointwise, uniform, conditional, or simultaneous, depending on the probability statement used in the construction. A confidence band controls coverage for an entire function or collection of parameter values, while a collection of separate pointwise intervals need not have the same simultaneous coverage. The distinction becomes important in regression curves, multiple comparisons, and other settings involving many related quantities.

Hypothesis testing

A frequentist test partitions the sample space into a rejection region and a non-rejection region. For a null hypothesis (H_0) and an alternative (H_1), a Type I error occurs when the test rejects (H_0) under a distribution contained in (H_0). A Type II error occurs when the test does not reject (H_0) under a distribution contained in (H_1).

The significance level is an upper bound on the Type I error probability,

[ \sup_{\theta\in\Theta_0} P_\theta(\text{reject }H_0) \leq \alpha. ]

The power function gives the rejection probability as a function of the parameter,

[ \pi(\theta)

P_\theta(\text{reject }H_0). ]

Power therefore describes both erroneous rejection under the null and successful rejection under alternatives, depending on the value of (\theta).

A p-value is a statistic whose distribution under the null satisfies

[ P_\theta(p\leq u)\leq u ]

for null parameter values and (0\leq u\leq1). In a continuous exact test, the p-value is uniformly distributed under a simple null hypothesis. Its observed value measures how extreme the chosen test statistic is relative to its null distribution. It is not the posterior probability that the null hypothesis is true, nor is its complement the probability that the alternative is true.

Fisherian significance testing and Neyman–Pearson testing employ related calculations but attach them to different formal structures. A Fisherian p-value summarizes evidence relative to a null model, whereas a Neyman–Pearson rule specifies behavior across repeated applications by fixing error rates and comparing alternatives. Modern practice often combines the vocabulary of both traditions, which accounts for several common interpretive ambiguities.

Sampling plans and stopping rules

The sampling plan is the probabilistic rule that determines how observations enter the data set. In a fixed-sample experiment, the number of observations is specified independently of their realized values. In a sequential experiment, continuation can depend on information accumulated during observation.

Because frequentist error probabilities are calculated over possible data sets generated by the full procedure, unadjusted repeated testing generally alters those probabilities. For example, testing after each additional observation until a conventional threshold is crossed creates a rejection probability different from that of a single fixed-sample test. Sequential methods account for this change by defining boundaries and error rates with respect to the sequential sampling process itself.

The relevance of stopping rules depends on the inferential construction. A likelihood function based on the final observations can be proportional under two different stopping plans, while the corresponding p-values or confidence sets can differ because their reference distributions include different unrealized outcomes. This contrast is associated with the likelihood principle, which states that proportional likelihood functions contain equivalent evidential information about the parameter. Frequentist procedures can satisfy the principle in particular models, but repeated-sampling calibration does not impose it generally.

Conditioning, nuisance parameters, and pivots

A nuisance parameter affects the sampling distribution without being the primary target of inference. Frequentist methods address nuisance parameters through profiling, conditioning, maximization of error probabilities, or statistics whose distributions do not depend on the nuisance quantity. Each construction produces a distinct repeated-sampling statement.

A pivotal quantity is a function of the data and parameter whose distribution is independent of unknown parameters. Inverting probability statements for a pivot can produce exact confidence intervals. Student’s (t)-statistic is a central example because its distribution under normal sampling does not depend on either the unknown mean or the unknown variance after standardization.

Conditioning restricts the reference distribution to outcomes sharing the value of an ancillary statistic, whose distribution does not depend on the parameter. Conditional and unconditional procedures can have the same aggregate error probability while distributing that error differently across ancillary subsets. This issue connects frequentist inference with principles of conditionality and with debates concerning which hypothetical repetitions are relevant to a realized experiment.

Model dependence and robustness

Frequentist guarantees are relative to the assumed data-generating family and sampling plan. Exact nominal coverage under one model does not imply the same coverage under misspecification. Dependence among observations, contamination, measurement error, and unmodeled selection can change the sampling distribution used to justify an estimator or test.

Robust statistics studies procedures whose behavior changes in controlled ways under deviations from an idealized model. The bootstrap approximates sampling distributions by resampling from the observed data or from a fitted model. Its frequentist interpretation rests on whether the resampling distribution consistently approximates the relevant sampling distribution, rather than on treating the resampled data sets as additional observations.

Large-sample approximations provide another form of model-based calibration. The central limit theorem can yield approximately normal sampling distributions even when the individual observations are not normal, although the quality of this approximation depends on distributional features and sample size. In irregular models, asymptotic limits can be non-normal or can depend on parameters in ways not captured by standard errors derived from regular likelihood theory.

Relation to other inferential frameworks

Bayesian inference combines a likelihood with a prior distribution to obtain a posterior distribution over unknown quantities. A Bayesian credible interval and a frequentist confidence interval can have identical numerical endpoints in some models, but their probability statements arise from different constructions. Bayesian calibration concerns posterior probability under the specified prior and model, while frequentist calibration concerns repeated-sampling behavior at fixed parameter values.

Likelihood inference emphasizes relative support conveyed by the observed likelihood without requiring a prior distribution or a complete repeated-sampling decision rule. Decision-theoretic methods can be frequentist or Bayesian depending on how risk is evaluated and how uncertainty about parameters is represented. These frameworks overlap mathematically, even though they differ in the interpretation assigned to probability and inferential output.