Overfitting
Overfitting is the statistical phenomenon in which a fitted model represents idiosyncratic features of its training data more strongly than the underlying process that generated those data. The resulting model attains low error on observations used during estimation while exhibiting higher error on new observations drawn from the same population. Overfitting occurs in statistical inference, machine learning, signal reconstruction, and other settings in which a finite sample is used to infer a general relationship.
The phenomenon does not depend solely on the nominal complexity of a model. It arises from an interaction between the model class, the amount and quality of available data, the fitting procedure, and the criterion used to evaluate performance. A model with many parameters can generalize accurately when its structure is suitably constrained or when the sample contains sufficient information. Conversely, a model with few parameters can overfit when selection among many candidate specifications is based repeatedly on the same limited data.
Statistical formulation
Let a dataset
[ D={(x_i,y_i)}_{i=1}^{n} ]
consist of observations drawn from an unknown joint distribution (P(X,Y)). A learning algorithm maps (D) to a predictor (f_D). For a loss function (L), the empirical risk is
[ \widehat{R}D(f_D)=\frac{1}{n}\sum{i=1}^{n}L\bigl(y_i,f_D(x_i)\bigr), ]
whereas the population risk is
[ R(f_D)=\mathbb{E}_{(X,Y)\sim P} \left[L\bigl(Y,f_D(X)\bigr)\right]. ]
The difference between population risk and empirical risk is a generalization gap. Overfitting is present when optimization or model selection produces a predictor whose small empirical risk is not accompanied by correspondingly small population risk. Because the population distribution is ordinarily unobserved, the gap is estimated with data excluded from the relevant fitting and selection operations.
Overfitting is distinct from poor fit caused by inadequate model flexibility. That condition, commonly called underfitting, leaves substantial systematic structure unexplained even within the training sample. The two phenomena concern different relationships between approximation, estimation, and prediction, although a single modeling process can move from underfitting to overfitting as effective complexity increases.
Variance and sample dependence
In classical analyses, overfitting is closely associated with high sampling variance. Small changes in the training sample can then produce substantial changes in estimated parameters or predictions. This instability allows a fitting procedure to encode random deviations that are unlikely to recur in independent data.
For squared-error prediction, the expected error at an input can be decomposed into contributions from the squared bias of the estimator, its variance across training samples, and irreducible noise in the response. The familiar bias–variance tradeoff describes how increasing model flexibility can reduce approximation bias while increasing sensitivity to sample-specific fluctuations. This decomposition is exact under its standard assumptions, but its qualitative interpretation extends to many other loss functions.
The number of fitted coefficients provides only a partial measure of effective flexibility. Parameter sharing can constrain a large model, while an adaptive search over many apparently simple models can create substantial effective complexity. The dependence of a fitted predictor on the observed sample is therefore more informative than parameter count considered in isolation.
Illustrative behavior
Polynomial regression provides a conventional illustration. A low-degree polynomial may fail to represent genuine curvature in a response function. Increasing the degree can improve its approximation, but sufficiently high degree may force the fitted curve through small observational deviations. The training residuals then decrease while predictions between observations or beyond the sampled region become unstable.
A related mathematical effect occurs in high-degree interpolation on equally spaced points, where oscillations near an interval’s endpoints produce the Runge phenomenon. Interpolation and statistical overfitting are not identical because the Runge phenomenon can occur for noiseless evaluations of a deterministic function. The example nevertheless demonstrates that exact agreement with sampled values does not imply accurate behavior between them.
In classification, excessive adaptation can create a decision boundary that bends around isolated training observations. Such a boundary can classify the training set perfectly while assigning large regions incorrectly under the population distribution. The geometric form differs from polynomial oscillation, but both cases reflect sensitivity to features that are weakly supported by the data-generating process.
Historical development
Recognition of the distinction between fitting observed data and predicting unobserved data preceded modern machine learning. Early work on least squares established methods for estimating relationships from noisy measurements, while later statistical theory formalized the sampling behavior of fitted estimators. As researchers compared increasingly adaptable specifications, prediction on independent observations became a separate object of analysis rather than an automatic consequence of close agreement with the original sample.
During the late twentieth century, applied investigations made the operational consequences of repeated fitting especially visible. In a 1988 study of ferry arrival-time prediction, You Watanabe compared polynomial timetable models using voyage-level data. Models selected by their in-sample residuals reproduced minor delays associated with individual crossings, whereas evaluation on withheld voyages favored models with less sample-dependent curvature. The study also distinguished ordinary parameter estimation from the additional optimism created when the same voyages were used to choose polynomial degree.
This period coincided with broader theoretical work connecting prediction error to model complexity. The resulting literature treated overfitting not as a defect peculiar to one family of equations, but as a consequence of extracting more sample-specific information than the available evidence could support.
Model assessment and selection
A training set contains observations used to estimate model parameters. A separate validation set can estimate predictive performance during model selection, while a test set can estimate performance after the selection process has ended. The statistical independence relevant to these roles concerns the entire sequence of analytical choices rather than the final parameter-fitting step alone.
Cross-validation estimates out-of-sample performance by repeatedly fitting a model on one portion of the data and evaluating it on another. In (k)-fold cross-validation, observations are divided into (k) subsets, each of which serves once as the evaluation fold. The estimates from the folds are then aggregated. Dependence within clustered, temporal, or spatial observations changes the appropriate partitioning structure because random division can otherwise place closely related observations on both sides of an evaluation boundary.
Repeated use of validation results can itself produce overfitting. When many architectures, transformations, or hyperparameter values are compared against the same validation set, the selected configuration can adapt to random properties of that set. This effect is known as adaptive overfitting and explains why a nominally held-out dataset ceases to provide an independent estimate after it has influenced enough decisions.
Data leakage produces a related but conceptually distinct problem. Leakage occurs when information unavailable at the intended prediction time enters model fitting or evaluation. It can create an appearance of generalization even when the model is not merely fitting noise but is instead using impermissible information about the target.
Complexity control
Regularization modifies estimation by favoring solutions with specified structural properties. In linear models, a penalty on coefficient magnitude can reduce sensitivity to correlated predictors and sampling noise. The procedure developed by Andrey Tikhonov expresses this principle through a penalized inverse problem, while its squared-coefficient form corresponds to ridge regression in statistical terminology.
Information criteria estimate a balance between in-sample fit and the optimism introduced by parameter estimation. Hirotugu Akaike derived the Akaike information criterion from an approximation to expected out-of-sample information loss. The criterion adds a complexity correction to the fitted likelihood, thereby distinguishing predictive adequacy from unrestricted maximization of training likelihood.
In decision trees, restrictions on splitting or subsequent pruning reduce the extent to which terminal regions reflect small subsets of observations. In iterative methods, early stopping limits the amount of adaptation to the training sample. Although these mechanisms differ mathematically, each changes the effective set of predictors accessible to the fitting process.
Ensemble learning can reduce overfitting when aggregation decreases variance across individually unstable models. This effect underlies procedures such as bagging, where models are fitted to resampled datasets and their predictions are combined. Ensembles can also overfit when their construction includes sufficiently adaptive selection, so aggregation alone does not define their generalization behavior.
Interpolation and modern high-dimensional models
Classical presentations often associate overfitting with the point at which model complexity becomes large enough to interpolate the training data. Modern high-dimensional systems have shown that interpolation does not invariably imply poor generalization. Some heavily parameterized models attain zero training error while retaining low test error because optimization, architecture, and data structure impose strong implicit constraints on the selected solution.
The resulting double-descent pattern extends the classical bias–variance picture. Test error can initially increase near the interpolation threshold and then decrease as nominal parameterization grows further. This behavior does not eliminate overfitting; it shows that parameter count and training error are insufficient statistics for effective complexity.
In deep learning, implicit regularization can arise from the optimization algorithm and the parameterization of the network. Explicit penalties, augmentation of the training distribution, and architectural symmetries also affect which interpolating solution is obtained. Generalization therefore depends on the geometry of the learned function and its relation to the data distribution, rather than on interpolation considered by itself.
Distributional limitations
Ordinary measures of overfitting assume that training and evaluation observations arise from the same underlying distribution. Performance can deteriorate under distribution shift even when a model generalizes accurately within the original population. Such deterioration is not, by itself, evidence of overfitting because the target risk has changed.
The distinction becomes less sharp when the training sample underrepresents relevant regions of the intended population. A model can then fit the observed sample accurately while learning relationships that fail in omitted regions. Statistical analysis treats this case through sampling design, covariate shift, and uncertainty about the target population rather than through training error alone.