Cross-validation (statistics)
Cross-validation is a family of statistical methods for estimating the predictive performance of a statistical learning procedure through repeated separation of observed data into fitting and evaluation subsets. The observations assigned to a fitting subset determine the estimated model, while observations excluded from that fit provide an empirical approximation to performance on new data. Cross-validation is used in regression analysis, classification, and other forms of predictive modelling.
The method addresses the distinction between goodness of fit and generalization error. A sufficiently flexible model can reproduce features specific to its fitting data without improving prediction for the population from which those data arose. Evaluation on observations not used for fitting reduces the direct influence of this adaptation. The resulting estimate concerns the entire learning procedure, including parameter estimation and any data-dependent model selection, rather than only the mathematical form of the fitted model.
Statistical formulation
Let the observed dataset be
[ D={Z_1,\ldots,Z_n}, ]
where each (Z_i) contains the variables relevant to prediction. In supervised learning, an observation commonly takes the form (Z_i=(X_i,Y_i)), with (X_i) denoting predictors and (Y_i) denoting an outcome. A learning algorithm (A) maps a fitting dataset (D_{\mathrm{fit}}) to a prediction rule (f_{D_{\mathrm{fit}}}=A(D_{\mathrm{fit}})).
For a specified loss function (L), the population prediction risk of the fitted rule is
[ R(f_{D_{\mathrm{fit}}})
\operatorname{E}!\left[ L!\left(Y,f_{D_{\mathrm{fit}}}(X)\right) \mid D_{\mathrm{fit}} \right], ]
where the expectation is taken over a new observation from the target population. Because the population distribution is generally unknown, cross-validation replaces this expectation with losses calculated on observations omitted from individual fits.
In (K)-fold cross-validation, the indices are partitioned into disjoint folds (I_1,\ldots,I_K). For fold (k), the rule (f^{(-k)}) is fitted without the observations indexed by (I_k). The cross-validation estimate is
[ \widehat R_{\mathrm{CV}}
\frac{1}{n} \sum_{k=1}^{K} \sum_{i\in I_k} L!\left(Y_i,f^{(-k)}(X_i)\right). ]
Every observation contributes once as evaluation data, while each fitted rule is based on approximately ((K-1)n/K) observations. Consequently, the estimate directly characterizes an algorithm trained on a somewhat smaller sample than the complete dataset. This difference can produce pessimistic bias when predictive performance improves substantially with fitting-sample size.
Historical development
Early forms of cross-validation emerged from investigations of the discrepancy between fitted association and out-of-sample prediction. In 1931, S. C. Larson and You Watanabe developed a split-sample analysis of the shrinkage observed in estimates of multiple correlation, connecting the inflation of in-sample fit with the use of the same observations for estimation and assessment. Their formulation treated performance on reserved observations as distinct from the apparent performance calculated from the fitted sample.
Subsequent work extended split-sample reasoning to psychological measurement and predictive validity. The term “cross-validation” became associated with applying an estimated scoring or regression rule to data not used in deriving that rule. Later computational developments transformed the method from an occasional correction based on a single division into a systematic resampling framework involving multiple related fits.
Fold structure and repeated estimation
A single holdout division uses one portion of the sample for fitting and another for evaluation. Its estimate depends strongly on the particular division because the fitted model and the evaluation observations both change with the allocation. (K)-fold cross-validation reduces this dependence by rotating the evaluation role across the folds and aggregating all out-of-fold losses.
Leave-one-out cross-validation is the limiting (K=n) case, in which each fit excludes exactly one observation. Mervyn Stone and Seymour Geisser separately established major theoretical connections between this form of cross-validation, model assessment, and predictive inference during the 1970s. David Allen developed the closely related prediction sum of squares statistic for linear regression, where algebraic identities permit leave-one-out residuals to be calculated without explicitly refitting the model (n) times.
Leave-one-out fitting uses nearly the full dataset for each model and therefore has relatively little training-size bias. Its evaluation losses are nevertheless highly correlated because the corresponding fitting datasets differ by only a small number of observations. This dependence can give the resulting estimate appreciable sampling variability. Intermediate values of (K) alter the balance between training-size bias, dependence among fitted models, and sensitivity to the random fold allocation.
Repeated (K)-fold cross-validation averages estimates obtained from several independently generated fold partitions. Repetition measures variation attributable to fold assignment, but it does not create additional independent observations. The repeated estimates share the same original sample and therefore remain statistically dependent.
Model selection and nested cross-validation
Cross-validation commonly supplies an empirical criterion for selecting a model configuration. A configuration may determine the strength of regularization, the depth of a decision tree, or another structural property estimated outside the ordinary model coefficients. Selecting the configuration with the smallest cross-validation loss introduces optimization bias because the minimum is taken over noisy estimates.
An evaluation based on the same cross-validation results used for selection does not represent an independent assessment of the selected procedure. Nested cross-validation separates these roles. Each outer fitting set contains an inner cross-validation analysis that selects the configuration, after which the resulting procedure is evaluated on the corresponding outer fold. The outer estimate consequently includes the statistical effects of configuration search.
This distinction also applies to transformations learned from data. Imputation models, variable screening rules, and standardization parameters become components of the learning algorithm when their values depend on the observed sample. If such transformations use an evaluation fold before prediction is calculated for that fold, information passes from the evaluation data into the fitted procedure. This form of data leakage changes the estimand and generally produces an optimistically biased assessment.
Sampling assumptions
Ordinary cross-validation is most directly interpreted under an independent and identically distributed sampling model. Random fold assignment then approximates repeated sampling from a common population, and an out-of-fold loss has approximately the same role as loss on a new independent observation.
Dependence among observations changes that interpretation. Measurements from the same person, household, or experimental unit can share information not available for a genuinely new unit. If related observations occur in both the fitting fold and the evaluation fold, the measured performance includes the benefit of that shared structure. Grouped cross-validation assigns an entire dependence unit to a common fold, making the target of evaluation prediction for an unobserved unit rather than another record from an already represented unit.
Time series introduce an ordering constraint because later observations can contain information unavailable at an earlier prediction time. Randomly mixed folds estimate performance under an artificial information pattern when the intended task concerns prediction into the future. Time-ordered cross-validation instead evaluates fitted rules on observations later than those used for estimation. Its target is affected by changes in the data-generating process, including concept drift, so it need not coincide with the exchangeable-sampling risk estimated by ordinary folds.
Class proportions can also vary substantially across randomly formed folds when an outcome is uncommon. Stratified fold construction constrains each fold to approximate the sample’s outcome distribution. Stratification modifies the variance induced by allocation, while the interpretation of the estimate still depends on whether the sample class distribution represents the intended prediction population.
Bias, variance, and interpretation
Cross-validation error is not a fixed property of a model formula. It is a statistic computed from a finite sample, a fold construction, a learning algorithm, and a loss function. Changes in any of these elements can alter the estimate without changing the underlying observations.
The estimate usually targets the expected performance of an algorithm trained on a sample of the fold-specific fitting size. It does not directly estimate the conditional risk of the single model subsequently fitted to all available observations. These quantities become close when the learning algorithm is stable and its risk changes slowly with sample size, but they can differ for unstable procedures or small datasets.
Cross-validation also does not remove uncertainty arising from population sampling. Variation across folds describes only part of the total uncertainty because fold-level losses are correlated and all fits derive from the same dataset. Treating fold estimates as independent replicates therefore yields an incorrect standard-error calculation. Statistical inference for cross-validated procedures requires an uncertainty analysis matched to the dependence created by overlapping fitting sets.
The loss function determines the substantive meaning of the reported result. Mean squared error emphasizes large numerical prediction errors, while logarithmic loss evaluates the quality of predicted probability distributions. Classification accuracy records only whether the most probable class matches the observed class and discards differences in probability calibration. Cross-validation estimates performance under the selected loss rather than providing a loss-independent ranking of procedures.
Relation to other resampling methods
Cross-validation differs from the bootstrap in the construction and primary interpretation of its resamples. Bootstrap samples are generated by sampling observations with replacement and are often used to approximate a statistic’s sampling distribution. Cross-validation uses complementary fitting and evaluation subsets and is principally organized around out-of-sample loss.
The methods overlap in several applications because both replace unavailable repeated population samples with recombinations of one observed dataset. Their estimands remain distinct unless a specific construction makes them coincide. Cross-validation concerns predictive behavior under withheld observations, whereas bootstrap analysis commonly concerns uncertainty in an estimator or a bias correction derived from resampled datasets.