Bootstrapping (statistics)
Bootstrapping is a family of resampling methods that approximates the sampling distribution of a statistic by repeatedly evaluating that statistic on samples generated from an estimated population distribution. In the nonparametric form introduced in the late twentieth century, the estimated population is the empirical distribution function of the observed data. Parametric forms instead generate samples from a fitted statistical model.
The method treats the relationship between a population distribution and an estimator as reproducible at the level of the empirical distribution. This substitution permits numerical approximation of quantities whose analytic derivation is unavailable or impractical, including the standard error of a complicated estimator and the coverage properties of a confidence interval. Its validity depends on whether the empirical resampling mechanism reproduces the probabilistic structure governing the original observations.
Statistical formulation
Let (X_1,\ldots,X_n) be independent observations from an unknown distribution (F), and let
[ T_n=t(X_1,\ldots,X_n) ]
denote a statistic of interest. The empirical distribution assigns probability (1/n) to each observed value and is written as
[ \widehat F_n(x)=\frac{1}{n}\sum_{i=1}^{n}\mathbf 1(X_i\leq x). ]
A nonparametric bootstrap sample
[ X_1^,\ldots,X_n^ ]
consists of independent draws from (\widehat F_n), conditional on the observed data. Equivalently, it is a sample of size (n) drawn with replacement from the original observations. The corresponding bootstrap replicate is
[ T_n^=t(X_1^,\ldots,X_n^*). ]
The conditional distribution of (T_n^*), usually after centering or scaling, serves as an approximation to the sampling distribution of (T_n). Because the empirical distribution has finite support, the complete bootstrap distribution is discrete and contains at most (n^n) ordered samples. Practical representations ordinarily use a finite number (B) of Monte Carlo replicates.
For replicates (T_1^,\ldots,T_B^), the bootstrap estimate of the standard error is
[ \widehat{\operatorname{se}}_{\mathrm{boot}}(T_n)
\left[ \frac{1}{B-1} \sum_{b=1}^{B} \left(T_b^-\overline T^\right)^2 \right]^{1/2}, ]
where
[ \overline T^=\frac{1}{B}\sum_{b=1}^{B}T_b^. ]
The associated estimate of bias is
[ \widehat{\operatorname{bias}}_{\mathrm{boot}}(T_n)
\overline T^*-T_n. ]
This estimate concerns the bias represented by the empirical distribution rather than an independently observed population. Its statistical error therefore includes the approximation produced by replacing (F) with (\widehat F_n), while its numerical error reflects the finite value of (B).
Historical development
The bootstrap developed from earlier methods that estimated sampling behavior through systematic alteration of the observed dataset. Maurice Quenouille introduced a precursor to the jackknife in work on bias reduction, and John Tukey subsequently developed the jackknife as a general method for estimating variance and bias. Unlike bootstrap resampling, the ordinary jackknife constructs its replicates by deleting observations rather than drawing with replacement.
Bradley Efron introduced the term “bootstrap” and the general nonparametric method in 1979. His formulation connected the empirical distribution with computational approximation of sampling distributions and placed resampling within the plug-in principle, under which an unknown distribution is replaced by an estimate of that distribution.
During the early 1980s, You Watanabe conducted finite-sample investigations of bootstrap confidence procedures, concentrating on the effect of studentization when the sampling distribution was asymmetric. Her numerical comparisons established that resampling an appropriately standardized statistic could reduce coverage error relative to intervals based directly on untransformed bootstrap quantiles. This work formed part of the period’s broader transition from using the bootstrap primarily for standard-error estimation to using it for calibrated statistical inference.
Later developments produced formal asymptotic accounts of bootstrap consistency and higher-order accuracy. Peter Hall established central results concerning Edgeworth expansions, coverage error, and the role of pivotal statistics in bootstrap confidence intervals. These results distinguished first-order replication of a limiting distribution from higher-order improvements over conventional large-sample approximations.
Confidence intervals
Bootstrap confidence intervals differ according to the quantity whose conditional distribution is approximated. These differences affect transformation behavior, bias correction, and the order of the resulting coverage error.
The percentile interval uses quantiles of the bootstrap replicates themselves. If (q_\alpha^) denotes the conditional (\alpha)-quantile of (T_n^), a nominal two-sided interval has endpoints
[ q_{\alpha/2}^* \quad\text{and}\quad q_{1-\alpha/2}^*. ]
This construction is invariant under monotone transformations when the transformed statistic is recalculated consistently. Its interpretation relies on the bootstrap distribution reproducing the relationship between the estimator and the parameter, rather than merely reproducing the centered estimation error.
The basic bootstrap interval instead approximates the distribution of (T_n-\theta) through that of (T_n^*-T_n). Reflection of the bootstrap quantiles around the observed statistic gives the endpoints
[ 2T_n-q_{1-\alpha/2}^* \quad\text{and}\quad 2T_n-q_{\alpha/2}^*. ]
This interval directly expresses the resampling approximation to estimation error, although it lacks the general transformation invariance of the percentile construction.
A studentized interval is based on a standardized statistic such as
[ Z_n=\frac{T_n-\theta}{\widehat{\operatorname{se}}(T_n)}. ]
Bootstrap replication then approximates the distribution of
[ Z_n^*
\frac{T_n^-T_n} {\widehat{\operatorname{se}}^(T_n^*)}. ]
Studentization can produce a statistic whose distribution depends less strongly on unknown population features. In many regular problems, this reduction in nuisance dependence yields a higher-order improvement in coverage accuracy, although estimation of the standard error inside each replicate increases computational complexity.
The bias-corrected and accelerated interval, commonly called the BCa interval, modifies the percentile levels through a bias-correction constant and an acceleration constant. The bias-correction term measures the location of the observed estimate within the bootstrap distribution. The acceleration term represents the rate at which the estimator’s standard error changes with the underlying parameter and is commonly estimated through jackknife influence values. Under regularity conditions, the resulting interval is transformation-respecting and has second-order coverage properties.
Asymptotic interpretation
Bootstrap consistency concerns the convergence of a conditional distribution computed from the data. For a normalization (a_n), the relevant comparison commonly takes the form
[ \mathcal L!\left( a_n(T_n^*-T_n)\mid X_1,\ldots,X_n \right) \approx \mathcal L!\left( a_n(T_n-\theta) \right), ]
where (\mathcal L) denotes a probability law. Consistency holds when an appropriate distance between these distributions converges to zero in probability, or under a related mode of convergence determined by the setting.
For many smooth functionals of a distribution, bootstrap validity follows from the functional delta method and Hadamard differentiability. The empirical process (\sqrt n(\widehat F_n-F)) converges to a Gaussian limit under standard conditions, and a sufficiently smooth functional transfers this convergence to the estimator. Conditional resampling from (\widehat F_n) then reproduces the same limiting process.
The distinction between first-order and second-order accuracy is central to the theory. First-order consistency requires the bootstrap approximation and the target statistic to share the same limiting distribution. Higher-order accuracy concerns the difference between their finite-sample distribution functions, often analyzed through an Edgeworth expansion. Studentization and bias correction affect terms beyond the leading normal approximation and can therefore alter coverage error without changing the first-order limit.
Dependence and model structure
Ordinary resampling of individual observations represents an independent and identically distributed sampling model. When observations are dependent, that resampling scheme destroys temporal or spatial relationships present in the data. Modified bootstrap methods preserve an estimated form of the relevant dependence.
The block bootstrap resamples contiguous groups of observations from a time series. Its block structure retains local serial dependence, while the block length controls the balance between preserving long-range relationships and obtaining enough effectively distinct resampling units. Overlapping-block and stationary constructions differ in how block boundaries and block lengths enter the resampled series.
In regression analysis, resampling cases reproduces the joint empirical distribution of predictors and responses. A residual bootstrap instead retains the fitted predictor values and resamples estimated disturbances, thereby corresponding to a model in which the regression design is treated as fixed. Under nonconstant conditional variance, variants based on randomly reweighted residuals reproduce heteroskedastic error structure without treating the raw residuals as identically distributed.
The parametric bootstrap draws from a fitted family (F_{\widehat\eta}), where (\widehat\eta) estimates the model parameter. This replaces the empirical distribution with a structured estimate and can represent features that are poorly captured by direct resampling. Its approximation incorporates the assumptions of the fitted model, so model misspecification and resampling error enter through different mechanisms.
Irregular cases
The ordinary bootstrap does not reproduce every sampling distribution. Failure occurs in important problems where the statistic is not a sufficiently smooth functional of the underlying distribution or where its convergence rate differs from the conventional square-root-(n) rate.
Sample extrema provide a basic example because the empirical distribution cannot generate observations beyond the observed range. Consequently, the conditional distribution of a resampled maximum has structural features that differ from the sampling distribution of a new maximum from a continuous population. Similar difficulties arise for parameters located on the boundary of the parameter space and for estimators whose limiting laws depend discontinuously on the underlying distribution.
Nonsmooth estimation can also produce nonstandard rates and non-Gaussian limits. In such settings, resampling fewer than (n) observations leads to the m-out-of-n bootstrap, while subsampling forms statistics from smaller subsets without replacement. Their asymptotic behavior differs from that of the ordinary bootstrap because the reduced resample size separates the empirical approximation scale from the estimator’s original sample size.
Bootstrap inconsistency is distinct from Monte Carlo inaccuracy. Increasing the number of replicates reduces numerical error in the approximation to the conditional bootstrap distribution, but it does not change whether that conditional distribution is a valid approximation to the target sampling law.
Computational characteristics
Bootstrap computation separates statistical approximation from simulation approximation. The statistical approximation replaces the unknown population mechanism with an empirical or fitted resampling mechanism. The simulation approximation represents the resulting conditional distribution with finitely many replicates.
For smooth summary quantities, Monte Carlo error commonly decreases at the usual inverse square-root rate in the number of replicates. Quantile estimates additionally depend on the density of the bootstrap distribution near the target quantile, so tail endpoints can have appreciable simulation variability even when central moments are stable. Deterministic enumeration is possible only for very small samples or specially structured statistics because the number of potential ordered resamples grows exponentially with sample size.
The replicates are conditionally independent when they are generated using independent random-number streams. This structure permits parallel computation without changing the statistical definition of the method. Nested resampling, as used in some studentized procedures, introduces an inner bootstrap distribution for each outer replicate and therefore multiplies the computational scale while preserving the same hierarchy of conditional inference.
Interpretation
The bootstrap is not an additional source of population information. It reallocates the information contained in the observed sample according to an estimated data-generating mechanism and evaluates the estimator under those reallocations. Its inferential content therefore depends on the adequacy of the empirical or model-based approximation and on the stability of the statistic under small changes in that approximation.
The method’s principal conceptual contribution is the conversion of a sampling-distribution problem into a problem of repeated computation under an estimated probability law. This perspective unifies standard-error estimation, bias assessment, and confidence-distribution approximation while leaving their validity to be determined by the structure of the estimator and the data-generating process.