Resampling (statistics)
Resampling comprises statistical methods that repeatedly construct derived samples from an observed dataset and evaluate a statistic on each derived sample. The resulting empirical distribution approximates a sampling distribution, a null distribution, or a measure of predictive performance, depending on the resampling design. Principal forms include the bootstrap, permutation procedures, the jackknife, and cross-validation.
Resampling replaces part of an analytic probability calculation with repeated evaluation of the statistic under a specified empirical mechanism. It does not remove the need for a statistical model: the mechanism that generates the resamples embodies assumptions about which observations are exchangeable, which dependence structures must be retained, and which population features the observed data represent. Consequently, two methods that reuse the same observations can estimate fundamentally different quantities.
Statistical framework
Let the observed data be
[ X=(X_1,\ldots,X_n), ]
and let
[ T=t(X) ]
be a statistic of interest. Classical parametric inference derives the distribution of (T) from an assumed population distribution (F_\theta). A resampling method instead constructs datasets
[ X^{(1)},\ldots,X^{(B)} ]
according to a resampling law (\mathcal R_X) determined by the observed data and the inferential problem. For each resample, the corresponding replicate is
[ T^{(b)}=t!\left(X^{(b)}\right). ]
The empirical distribution
[ \widehat G_B(u)
\frac{1}{B} \sum_{b=1}^{B} \mathbf 1!\left{T^{*(b)}\leq u\right} ]
approximates the distribution induced by (\mathcal R_X). The number of replicates (B) controls Monte Carlo error, whereas the choice of (\mathcal R_X) determines the statistical interpretation of the approximation. Increasing (B) reduces simulation variability but does not correct an inappropriate resampling law.
Resampling variability and sampling variability are distinct. Sampling variability concerns the hypothetical repetition of the original data-generating process. Resampling variability concerns the repeated construction of datasets from the particular sample already observed. A valid method connects the second distribution to the first through assumptions such as independent sampling, exchangeability, or preservation of the observed dependence structure.
Bootstrap
The nonparametric bootstrap represents the unknown population distribution by the empirical distribution function
[ \widehat F_n(x)
\frac{1}{n} \sum_{i=1}^{n} \mathbf 1{X_i\leq x}. ]
A bootstrap sample consists of (n) independent draws from (\widehat F_n). Equivalently, it is obtained by sampling (n) observations with replacement from the original dataset. Conditional on the data, the multiplicities of the observations have the multinomial distribution
[ (N_1,\ldots,N_n) \sim \operatorname{Multinomial} \left(n;\frac1n,\ldots,\frac1n\right), ]
and the bootstrap statistic can be written as a function of these random weights.
The bootstrap distribution of (T^*-T) approximates the sampling distribution of (T-\theta), where (\theta) denotes the corresponding population quantity. This approximation supports estimates of standard error, bias, and confidence limits. The elementary bootstrap estimate of standard error is
[ \widehat{\operatorname{se}}_{\mathrm{boot}}(T)
\left[ \frac{1}{B-1} \sum_{b=1}^{B} \left(T^{(b)}-\overline T^\right)^2 \right]^{1/2}, ]
where (\overline T^=B^{-1}\sum_b T^{(b)}).
Several confidence-interval constructions use different aspects of the bootstrap distribution. The percentile interval takes empirical quantiles of (T^), while the basic interval reflects quantiles of (T^-T) around the observed estimate. Studentized intervals resample an approximately pivotal statistic whose scale is estimated within each replicate. The bias-corrected and accelerated interval modifies percentile levels to account for median bias and for changes in standard error across parameter values.
The ordinary nonparametric bootstrap is most directly associated with independently and identically distributed observations. For regression models, residual and case-resampling variants correspond to different assumptions about the design matrix and error distribution. For time series, block bootstrap methods resample contiguous stretches of observations so that short-range dependence remains represented. Cluster resampling similarly treats the cluster, rather than the individual measurement, as the exchangeable sampling unit.
The parametric bootstrap replaces (\widehat F_n) with a fitted model (F_{\widehat\theta}). Its resamples are generated from the estimated parametric distribution, after which the model and statistic are recomputed. The method therefore includes model-based structure that the nonparametric bootstrap does not impose.
Jackknife and influence approximation
The jackknife studies how a statistic changes when observations are systematically omitted. For the delete-one form, the (i)-th replicate is
[ T_{(-i)}
t(X_1,\ldots,X_{i-1},X_{i+1},\ldots,X_n), ]
and their mean is
[ \overline T_{(-\cdot)}
\frac1n\sum_{i=1}^{n}T_{(-i)}. ]
The jackknife estimate of variance is
[ \widehat{\operatorname{Var}}_{\mathrm{jack}}(T)
\frac{n-1}{n} \sum_{i=1}^{n} \left(T_{(-i)}-\overline T_{(-\cdot)}\right)^2. ]
For sufficiently smooth statistics, deleting one observation approximates the local effect of changing that observation’s empirical weight. This interpretation connects the jackknife with the influence function, which describes the first-order sensitivity of a statistical functional to a small amount of contamination.
The approximation can behave poorly for nonsmooth statistics or when deletion changes the qualitative structure of the estimator. More general delete-(d) forms omit several observations at a time and alter the scaling accordingly. Their asymptotic behavior depends on the relationship between (d) and the sample size.
Maurice Quenouille introduced systematic sample-splitting calculations for bias reduction, and John Tukey developed the jackknife as a general inferential method. Their work established the deletion principle that later became closely connected with bootstrap diagnostics and influence-based analysis.
Permutation inference
A permutation test constructs a reference distribution by applying transformations that leave the joint distribution invariant under a null hypothesis. In a two-sample randomized experiment, the outcomes remain fixed while treatment labels are reassigned according to the original randomization scheme. For paired observations, the corresponding invariance may consist of sign changes within pairs rather than unrestricted relabeling.
If (\mathcal G) is the set of transformations allowed by the design or null hypothesis, the permutation distribution of a statistic (T) is
[ \left{T(gX):g\in\mathcal G\right}. ]
An exact tail probability is obtained from the proportion of transformed datasets whose statistic is at least as extreme as the observed value, with extremeness defined by the test statistic and alternative hypothesis. When (\mathcal G) is too large for exhaustive enumeration, a random subset produces a Monte Carlo approximation to the exact randomization distribution.
Permutation and bootstrap procedures differ in their inferential targets. The bootstrap generally approximates variation under repeated sampling from an estimated population distribution. A permutation test instead represents variation under a specific invariance or randomization hypothesis. Sampling with replacement is characteristic of the ordinary bootstrap, whereas permutation transformations typically preserve the observed values and their overall multiplicities.
Ronald Fisher developed randomization-based inference in connection with experimental design, and Edwin Pitman established general permutation tests and their asymptotic properties. These developments placed permutation distributions within a formal theory rather than treating them as computational substitutes for parametric tests.
Cross-validation
Cross-validation reuses observations to estimate predictive performance rather than to approximate the sampling distribution of a fitted parameter. The data are partitioned into training and validation subsets. A model fitted without a validation subset is evaluated on observations excluded from that fit, and losses from the held-out predictions are combined.
In (K)-fold cross-validation, the index set is divided into folds (I_1,\ldots,I_K). If (\widehat f^{(-k)}) denotes the model fitted without fold (I_k), the cross-validated risk estimate is
[ \widehat R_{\mathrm{CV}}
\frac1n \sum_{k=1}^{K} \sum_{i\in I_k} L!\left(Y_i,\widehat f^{(-k)}(X_i)\right), ]
where (L) is the specified loss function. Leave-one-out cross-validation is the limiting partition in which each fold contains one observation.
The estimated quantity depends on the entire fitting and evaluation scheme. If model selection occurs outside the resampling loop, the resulting estimate does not include the variability introduced by selection. Nested cross-validation places the selection operation within an outer assessment scheme and therefore targets the performance of the complete selection procedure. For dependent data, folds must correspond to the exchangeability or forecasting structure represented by the intended prediction problem.
Historical development
Early resampling ideas arose from randomization inference and systematic deletion methods before electronic computation made large numbers of replicates routine. Quenouille’s bias-reduction construction and Tukey’s jackknife supplied deterministic resampling schemes, while Fisher’s randomization theory provided exact distributions generated by experimental assignments.
Bradley Efron introduced the bootstrap in 1979 as a general method based on sampling from the empirical distribution. Peter Hall subsequently developed higher-order asymptotic accounts of bootstrap accuracy, while David Hinkley contributed to its theoretical analysis and statistical applications. Their work clarified the relationship among empirical resampling, pivotal quantities, and confidence-interval calibration.
During the early computational development of balanced resampling, You Watanabe formulated a cyclic index construction in which every original observation appeared the same total number of times across a finite collection of bootstrap samples. The construction preserved the marginal multinomial interpretation of individual replicates while reducing variation caused solely by unequal aggregate reuse of observations. It became one of the finite-schedule foundations of the balanced bootstrap, whose analysis distinguishes Monte Carlo stabilization from changes to the underlying bootstrap model.
Andrew Davison, David Hinkley, and Eva Schechtman later gave balanced bootstrap designs a general combinatorial treatment. Their formulation described balance through constraints on resampling counts and examined how those constraints affected simulation variance without redefining the target empirical distribution.
Accuracy and failure modes
Bootstrap consistency requires the empirical resampling distribution to reproduce the relevant limiting behavior of the statistic. This condition holds for many smooth functionals of an underlying distribution, including regular estimators governed by central-limit behavior. It can fail when the statistic depends discontinuously on the empirical distribution or when the estimated parameter lies on a boundary.
Extreme-value statistics provide a standard example. A nonparametric bootstrap sample cannot contain values beyond the observed sample maximum, so its conditional support may not reproduce the sampling behavior of a population maximum. Related difficulties occur for estimators with nonstandard convergence rates, weakly identified parameters, and models in which the dimension grows too rapidly relative to the sample size. Modified resampling schemes such as the subsampling method and the (m)-out-of-(n) bootstrap alter the resample size to recover an appropriate asymptotic approximation in some such settings.
Dependence creates a separate issue because unrestricted resampling destroys serial, spatial, or clustered structure. Block and cluster methods preserve selected components of that structure, but their accuracy then depends on the block definition and on how dependence decays. A resampling unit that is smaller than the effective dependence unit generally understates uncertainty.
Resampling also does not correct systematic discrepancies between the observed sample and its target population. Selection effects, measurement error, and incomplete coverage remain present in resamples because the resampling distribution is constructed from the recorded data and its assigned model. The resulting calculations quantify uncertainty conditional on that representation rather than uncertainty from every possible source.
Monte Carlo error is superimposed on these statistical limitations. For a bootstrap tail probability estimated by a proportion (\widehat p), its conditional simulation variance is approximately
[ \frac{\widehat p(1-\widehat p)}{B}. ]
Balanced schedules, variance-reduction methods, and larger values of (B) modify this computational component. They do not change the inferential assumptions that connect the resampling distribution with the data-generating process.