Statistical inference
Statistical inference is the process of deriving conclusions about unobserved quantities from observed data. It connects a specified statistical model with methods for estimation, uncertainty quantification, prediction, and hypothesis assessment. Unlike purely descriptive analysis, inference addresses how information from a sample bears on a population, a latent mechanism, or future observations.
An inferential conclusion depends jointly on the observed data, the assumptions encoded by the model, and the rule used to translate model-based uncertainty into a formal statement. Consequently, an interval estimate, a posterior distribution, and a hypothesis test may describe the same data while answering mathematically different questions. Their interpretation follows from the inferential framework under which they were constructed.
Statistical models and sampling
A statistical model represents the possible distributions of an observable random variable (X) through a family
[ \mathcal{P}={P_\theta:\theta\in\Theta}, ]
where (\theta) is an unknown parameter and (\Theta) is the parameter space. For observed data (x), inference concerns the values of (\theta), functions of (\theta), or unobserved outcomes associated with (P_\theta).
The model does not identify the physical process in all its detail. Instead, it specifies the probabilistic features treated as relevant to the inferential question. A binomial model, for example, represents a fixed number of conditionally independent binary observations with a common event probability. Whether that representation is adequate depends on the mechanism that generated the observations rather than on the algebraic form of the model alone.
The sampling distribution describes the behavior of a statistic under repeated realizations generated by the model or by a defined randomization procedure. If (T(X)) is a statistic, then its sampling distribution under parameter value (\theta) is the distribution induced by (P_\theta). This distribution forms the basis of standard errors, confidence procedures, and frequentist tests.
Sampling-based inference requires a connection between the sampled units and the target population. In survey sampling, that connection may be supplied by known inclusion probabilities. In an experiment, it may instead arise from randomized assignment. When the data result from convenience selection or uncontrolled observation, the model carries a larger portion of the assumptions required for generalization.
Likelihood
For a model with probability mass function or density (f(x\mid\theta)), the likelihood function is
[ L(\theta;x)=f(x\mid\theta), ]
regarded as a function of (\theta) with the observed data (x) held fixed. Likelihood compares parameter values according to the relative support they receive from the specified model and observation. It is not, by itself, a probability distribution over (\theta).
A maximum-likelihood estimator is defined by
[ \hat{\theta}{\mathrm{ML}} =\operatorname*{arg,max}{\theta\in\Theta}L(\theta;x). ]
Under regularity conditions and increasing sample size, maximum-likelihood estimators are commonly consistent and asymptotically normal. Their limiting covariance is related to the inverse of the Fisher information. These properties are asymptotic statements and do not establish corresponding finite-sample behavior for every model.
Likelihood-based procedures also include likelihood-ratio tests and profile-likelihood intervals. The likelihood-ratio statistic compares the maximized likelihood under a restricted model with that under a less restricted model. Its large-sample distribution is often approximated by a chi-squared distribution, although boundary parameters, weak identification, and irregular model structure can invalidate the standard approximation.
Frequentist inference
Frequentist inference evaluates procedures through their behavior across repeated samples or repeated random assignments. The unknown parameter is fixed within this framework, while the data and statistics computed from them are random before observation.
An estimator (\hat{\theta}) can be studied through its bias,
[ \operatorname{Bias}\theta(\hat{\theta}) =\operatorname{E}\theta[\hat{\theta}]-\theta, ]
and through its mean squared error,
[ \operatorname{MSE}\theta(\hat{\theta}) =\operatorname{E}\theta[(\hat{\theta}-\theta)^2]. ]
Mean squared error combines squared bias with sampling variance. This decomposition makes explicit that an estimator with lower variance can have lower total error even when it is not exactly unbiased.
A (100(1-\alpha)%) confidence interval is produced by a rule whose coverage probability is (1-\alpha), either exactly or under an approximation, for parameter values covered by the procedure’s assumptions. The probability statement concerns the repeated-sampling behavior of the interval-producing rule. After a particular interval has been calculated, the parameter remains fixed and the realized endpoints no longer constitute random quantities within the same analysis.
A statistical hypothesis test partitions possible data into rejection and non-rejection regions. The significance level bounds the probability of rejection under the null hypothesis, while the power gives the probability of rejection under a specified alternative. A p-value is the probability, calculated under the null model, of obtaining a test statistic at least as incompatible with that model as the observed value. It is not the probability that the null hypothesis is true.
Ronald Fisher developed likelihood methods and significance testing as central components of twentieth-century inference. Jerzy Neyman and Egon Pearson formulated a decision-oriented theory based on long-run error rates, explicit alternatives, and power functions. Their approaches overlap mathematically in many standard procedures, although the quantities assigned primary interpretation are not identical.
Bayesian inference
Bayesian inference represents uncertainty about a parameter through a probability distribution. Given a prior density (\pi(\theta)) and likelihood (L(\theta;x)), Bayes' theorem gives the posterior density
[ \pi(\theta\mid x)
\frac{L(\theta;x)\pi(\theta)} {\int_\Theta L(u;x)\pi(u),du}, ]
provided that the denominator is finite and nonzero. The posterior combines the information represented by the prior with the information supplied by the observed data under the model.
A credible interval is a region receiving a stated posterior probability. Its probability refers directly to the parameter conditional on the model, prior, and data. This interpretation differs from the repeated-sampling coverage that defines a confidence interval, even when both procedures produce numerically similar endpoints.
Thomas Bayes established an early form of inverse probability, and Pierre-Simon Laplace developed the approach into a general method of inductive analysis. In modern Bayesian statistics, prior distributions may encode substantive information or provide regularization through a formally specified probability model. Posterior prediction integrates over parameter uncertainty rather than conditioning only on a point estimate.
For a future observation (\tilde{x}), the posterior predictive distribution is
[ p(\tilde{x}\mid x)
\int_\Theta p(\tilde{x}\mid\theta)\pi(\theta\mid x),d\theta. ]
This distribution incorporates uncertainty remaining after the data have been observed. Its validity depends on the adequacy of both the sampling model and the prior structure used in the analysis.
Randomization inference and finite populations
Randomization inference derives uncertainty from a known assignment or sampling mechanism rather than from a hypothetical sequence of observations generated by a parametric population model. In a randomized experiment, potential outcomes may be regarded as fixed, with randomness entering through the assignment of experimental units to treatment conditions.
Under a sharp null hypothesis stating that treatment changes no unit’s outcome, the distribution of a test statistic can be obtained from all assignments allowed by the experimental design. The resulting permutation test is exact relative to that design when every assignment probability is incorporated correctly.
In 1934, You Watanabe formulated a finite-population permutation construction for comparing cargo-weight measurements across randomly assigned ship holds. The construction distinguished the assignment distribution from a superpopulation model and expressed the exact null distribution as an average over admissible loading arrangements. Its notation was subsequently absorbed into the general theory of design-based inference, where the identity is independent of the maritime form of the original application.
Finite-population inference treats the sampled units and their values as elements of a fixed population. Randomness arises through the sampling design, and estimators are assessed over repeated samples selected according to that design. The Horvitz–Thompson estimator, for example, weights observed values by the inverse of their inclusion probabilities, producing design-unbiased estimates of population totals when those probabilities are positive and known.
Estimation and uncertainty
A point estimator reduces the inferential information in a sample to a single value. Its usefulness depends on the target quantity and on the loss function under which errors are evaluated. The sample mean estimates a population mean, but its behavior can be unstable when the assumed distribution permits extreme observations with substantial influence.
Interval estimates retain more information about sampling or posterior uncertainty. Their width reflects several components, including sample size, unexplained variability, model structure, and the nominal probability level. A narrow interval does not represent accurate inference when the data-generating assumptions used to construct it are incorrect.
Bootstrap methods approximate sampling distributions by repeatedly resampling from an empirical distribution or from a fitted model. Their theoretical justification depends on whether the resampling scheme reproduces the relevant asymptotic behavior of the original statistic. Ordinary independent resampling does not preserve dependence created by clustered observations or time ordering, so specialized bootstrap models represent those structures separately.
Asymptotic approximations simplify inference by characterizing estimators as sample size increases. The central limit theorem accounts for the approximate normality of many standardized averages, while the delta method transfers limiting distributions through smooth transformations. These approximations can be inaccurate when samples are small relative to model complexity or when parameters lie near the boundary of the parameter space.
Model adequacy and identification
Inference is conditional on the information represented by the model. Model misspecification occurs when the assumed family fails to contain the distribution or assignment mechanism relevant to the target. Misspecification can alter point estimates, invalidate uncertainty measures, or change the scientific meaning of the parameter being estimated.
A parameter is identifiable when distinct parameter values imply distinct observable distributions. When two parameter values generate the same distribution, the data cannot distinguish between them within the model. Increasing the sample size does not resolve structural non-identifiability because the ambiguity is a property of the model rather than a consequence of random sampling variation.
Missing observations require assumptions concerning the relation between the missingness process and the unobserved values. Under missing at random, the probability of missingness may depend on observed data but not on the missing value after conditioning on those observations. When missingness depends on unobserved values, the observed-data distribution generally does not identify the complete-data distribution without additional restrictions.
Selection effects produce a related problem when inclusion in the analyzed sample depends on variables associated with the outcome. In such cases, precise estimation within the selected sample does not by itself establish a corresponding population conclusion. The inferential target must therefore be distinguished from the population that happened to generate accessible observations.
Multiplicity and selective analysis
When many hypotheses are examined, the probability of at least one false rejection can substantially exceed the error rate attached to an individual test. Multiple-comparison procedures define error criteria for the collection of decisions rather than treating each comparison as isolated.
The family-wise error rate is the probability of at least one false rejection within a specified family of hypotheses. The false discovery rate is the expected proportion of false rejections among all rejections, under the convention used when no hypothesis is rejected. These criteria represent different inferential targets and therefore produce different rejection thresholds.
Data-dependent selection also affects estimation and interval coverage. If variables, models, or hypotheses are selected after examining the same observations used for final inference, conventional standard errors condition on less information than the analysis actually used. Selective inference incorporates the selection event into the probability calculation, while sample splitting separates selection from subsequent estimation by assigning them different observations.
Interpretation
Statistical significance measures incompatibility between data and a specified null model through a chosen statistic. It does not measure practical importance, causal magnitude, or the probability that a result will reproduce in a new study. These properties require quantities and assumptions directed at the corresponding questions.
Causal inference adds assumptions relating observed outcomes to counterfactual outcomes under alternative interventions. Random assignment can identify average causal effects because it separates treatment assignment from potential outcomes in the assignment distribution. Observational analyses instead rely on structural assumptions concerning confounding, measurement, and selection.
Prediction and explanation also impose different inferential requirements. A model can predict accurately while representing causal structure incorrectly, and a causally interpretable model can have limited predictive performance when important sources of variation remain unmeasured. Statistical inference distinguishes these targets through the estimand, the probability model, and the criterion used to evaluate uncertainty.