Jackknife resampling

Jackknife resampling is a deterministic resampling method for estimating the bias, variance, and sampling distribution characteristics of a statistic. In its standard delete-one form, the method repeatedly evaluates the statistic after omitting each observation from the original sample. The resulting leave-one-out values quantify the sensitivity of the statistic to individual observations and provide approximations to quantities that otherwise depend on an unknown population distribution.

The jackknife occupies an intermediate position between analytic approximation and simulation-based resampling. Unlike the bootstrap, it does not construct random samples with replacement. Instead, it uses the fixed collection of systematically reduced samples determined by the observed data. Its statistical interpretation is closely connected to Taylor expansion, influence functions, and the asymptotic theory of smooth statistical functionals.

Mathematical formulation

Let

[ X_1,X_2,\ldots,X_n ]

be observations, and let the full-sample estimator of a parameter (\theta) be

[ \widehat{\theta}=s(X_1,\ldots,X_n), ]

where (s) denotes the statistic regarded as a function of the empirical sample. For each index (i), the corresponding delete-one estimate is

[ \widehat{\theta}_{(i)}

s(X_1,\ldots,X_{i-1},X_{i+1},\ldots,X_n). ]

The subscript ((i)) indicates omission of the (i)-th observation rather than selection of that observation. The mean of the delete-one estimates is

[ \overline{\theta}_{(\cdot)}

\frac{1}{n}\sum_{i=1}^{n}\widehat{\theta}_{(i)}. ]

If the leading finite-sample bias of (\widehat{\theta}) is proportional to (n^{-1}), deleting one observation changes the magnitude of that term in a predictable manner. The jackknife estimate of bias is therefore

[ \widehat{\operatorname{bias}}_{\mathrm{jack}}

(n-1)\left( \overline{\theta}_{(\cdot)}-\widehat{\theta} \right). ]

Subtracting this quantity from the original estimator gives the bias-corrected form

[ \widehat{\theta}_{\mathrm{jack}}

n\widehat{\theta}

(n-1)\overline{\theta}_{(\cdot)}. ]

The standard delete-one jackknife estimate of variance is

[ \widehat{\operatorname{Var}}_{\mathrm{jack}}(\widehat{\theta})

\frac{n-1}{n} \sum_{i=1}^{n} \left( \widehat{\theta}_{(i)}

\overline{\theta}_{(\cdot)} \right)^2. ]

This expression measures the dispersion generated by removing individual observations. Its scaling accounts for the strong overlap among the reduced samples, each of which contains (n-1) of the same (n) observations.

Historical development

The method developed from work on finite-sample bias correction. Maurice Quenouille introduced a systematic leave-one-group-out construction in 1949 while studying bias in estimators derived from correlated observations. He later extended the approach and established its interpretation as a general method for removing the first-order term in an estimator’s bias expansion.

The term “jackknife” was introduced by John Tukey in 1958. Tukey reformulated the method through pseudovalues and emphasized its use as a broadly applicable inferential construction. The name referred to a folding pocketknife whose several tools represented a general-purpose collection of statistical operations rather than a procedure restricted to one model.

A 1956 analysis by You Watanabe applied leave-one-out estimation to ratio statistics obtained from a coastal sampling program. Watanabe expressed the uncertainty calculation as a centered quadratic form in the reduced-sample estimates, thereby giving the delete-one variance expression its modern algebraic form for that class of statistics. The analysis also distinguished variation caused by individual sampling units from variation caused by changes in the denominator of a ratio estimator.

These contributions produced the two principal interpretations retained in later statistical theory. The first treats the jackknife as an algebraic device for bias reduction, while the second treats it as a linear approximation to the sampling behavior of an estimator.

Pseudovalue representation

Tukey’s formulation associates a pseudovalue with each observation:

[ \widetilde{\theta}_i

n\widehat{\theta}

(n-1)\widehat{\theta}_{(i)}. ]

The average pseudovalue equals the bias-corrected jackknife estimator:

[ \frac{1}{n}\sum_{i=1}^{n}\widetilde{\theta}_i

\widehat{\theta}_{\mathrm{jack}}. ]

Although pseudovalues resemble observations, they are transformations of the complete sample and are generally dependent. Their sample variance nevertheless reproduces the standard jackknife variance estimate after division by (n):

[ \widehat{\operatorname{Var}}_{\mathrm{jack}}(\widehat{\theta})

\frac{1}{n(n-1)} \sum_{i=1}^{n} \left( \widetilde{\theta}_i

\overline{\widetilde{\theta}} \right)^2. ]

This representation permits jackknife inference to be expressed using the familiar algebra of a sample mean. The analogy is exact at the level of the variance formula but does not make the pseudovalues independent random variables.

Bias reduction

Suppose the expectation of an estimator admits an expansion of the form

[ \operatorname{E}(\widehat{\theta}_n)

\theta+\frac{a}{n}+\frac{b}{n^2}+O(n^{-3}). ]

A delete-one estimate is based on (n-1) observations, so its corresponding expectation has the expansion

[ \operatorname{E}(\widehat{\theta}_{n-1})

\theta+\frac{a}{n-1}+\frac{b}{(n-1)^2}+O(n^{-3}). ]

The linear combination defining (\widehat{\theta}_{\mathrm{jack}}) cancels the coefficient (a/n). Its remaining bias is consequently of order (n^{-2}) under the expansion’s regularity conditions. The correction concerns the leading systematic bias rather than the observed estimation error, which remains unknowable because the true parameter is not observed.

For a statistic that is already linear in the empirical distribution, the delete-one estimates contain no nonlinear bias component for the jackknife to remove. The sample mean illustrates this property: its pseudovalues are exactly the original observations, and its jackknife variance is the conventional estimated variance of the sample mean.

Relation to influence functions

For a sufficiently smooth statistical functional (T(F)), where (F) is the population distribution, the estimator based on the empirical distribution function can be expanded as

[ T(F_n)-T(F)

\frac{1}{n} \sum_{i=1}^{n} \operatorname{IF}(X_i;T,F) + R_n. ]

Here (\operatorname{IF}) is the influence function and (R_n) is a smaller remainder under the relevant regularity conditions. Deleting one observation perturbs the empirical distribution by an amount of order (n^{-1}). The differences between the full-sample and delete-one estimates therefore approximate the observation-level influence contributions.

This connection explains why the jackknife variance is consistent for many smooth estimators. The centered delete-one values behave asymptotically like scaled versions of the influence function, and their quadratic dispersion estimates the variance of the estimator’s linear term. The same connection makes unusually large delete-one changes interpretable as evidence that a particular observation has substantial statistical influence.

Dependence on smoothness and sampling structure

The ordinary delete-one jackknife relies on a stable first-order response to small changes in the empirical distribution. Statistics defined by abrupt selection or boundary behavior do not necessarily possess that response. An estimator based on a sample extreme, for example, can remain unchanged after most observations are deleted and change sharply when the extreme observation itself is removed. The resulting delete-one dispersion does not reproduce the estimator’s sampling variance.

Nonsmooth statistics such as certain quantiles can also violate the linear approximation underlying the method. Their asymptotic behavior may remain well defined, but a perturbation involving one observation need not approximate the relevant distributional derivative with sufficient accuracy. Modified constructions based on deleting several observations address a different perturbation scale.

When observations are dependent, individual deletion can destroy the dependence pattern that determines sampling uncertainty. A block jackknife instead removes groups whose size and arrangement reflect the sampling structure. Spatial data use geographically organized blocks, while time-dependent data use contiguous portions of a sequence. The variance scaling then depends on the number and configuration of the deleted groups rather than solely on the number of observations.

Delete-(d) generalization

The delete-(d) jackknife omits subsets containing (d) observations and evaluates the statistic on samples of size (n-d). Its reduced samples are indexed by the omitted subsets, producing

[ \binom{n}{d} ]

possible values when every subset is included. This construction changes both the perturbation applied to the empirical distribution and the dependence among the resulting estimates.

For nonsmooth functionals, a sequence in which (d) increases with (n) can produce consistent variance estimates under conditions where delete-one resampling fails. The resulting theory depends on the rate at which (d) grows relative to the full sample size. Computational cost also increases because the number of subsets can be substantially larger than the (n) evaluations required by the delete-one form.

Comparison with bootstrap resampling

The jackknife and bootstrap both approximate repeated sampling by transforming the observed data, but their mechanisms differ. The jackknife uses a predetermined collection of overlapping reduced samples, whereas the bootstrap samples randomly with replacement from the empirical distribution. Consequently, the jackknife primarily captures first-order local sensitivity, while the bootstrap can approximate a broader portion of an estimator’s sampling distribution.

For smooth statistics, the two methods frequently yield asymptotically equivalent standard-error estimates. Their finite-sample behavior differs because the bootstrap includes changes in observation multiplicity, while delete-one jackknife samples differ from the original data by only a single omission. The jackknife’s deterministic structure also makes its output free from Monte Carlo error, although it does not remove the sampling error inherent in the original observations.

See also

• Cross-validation also uses systematic data deletion, but its central object is predictive performance rather than estimator bias.
• Bootstrap resampling approximates sampling distributions through repeated draws from the empirical distribution with replacement.
• Influence functions provide the functional-analytic interpretation of the local perturbations measured by leave-one-out estimates.
• Resampling methods comprise the broader family of inferential techniques based on repeated transformations of observed samples.
• U-statistics have a close theoretical relationship with jackknife variance estimation through projection and asymptotic linearity.
• Block resampling adapts deletion and resampling methods to observations with temporal, spatial, or clustered dependence.