Curve fitting
Curve fitting is the construction of a mathematical function whose values correspond, according to a specified criterion, to a collection of observed data. The fitted function may represent an assumed physical relation, summarize the systematic component of a sample, or provide interpolation between measured points. Its parameters are determined from the data rather than fixed independently of them.
A curve fit does not require the fitted object to be curved in the geometric sense. A straight line is a curve within this terminology, as is a surface fitted to observations with several explanatory variables. The broader statistical formulation is known as regression analysis, while exact passage through every observation is treated as interpolation.
Mathematical formulation
For observations
[ (x_1,y_1),\ldots,(x_n,y_n), ]
a parametric model represents the response as
[ y_i=f(x_i;\boldsymbol{\theta})+\varepsilon_i, ]
where (f) is a chosen family of functions, (\boldsymbol{\theta}) is a parameter vector, and (\varepsilon_i) denotes the discrepancy between the observation and the model value. The fitted parameter vector minimizes or otherwise optimizes an objective function constructed from these discrepancies.
The most common objective is the residual sum of squares,
[ S(\boldsymbol{\theta}) =\sum_{i=1}^{n} \left[y_i-f(x_i;\boldsymbol{\theta})\right]^2. ]
Its minimizer,
[ \widehat{\boldsymbol{\theta}} =\operatorname*{arg,min}_{\boldsymbol{\theta}} S(\boldsymbol{\theta}), ]
defines the ordinary least-squares fit. Squaring gives large residuals greater influence than small residuals and produces a differentiable criterion for many model families. Under independent Gaussian measurement errors with constant variance, the same parameter estimate is obtained by maximum likelihood estimation.
When observations have unequal variances, the criterion can assign a weight (w_i) to each squared residual:
[ S_w(\boldsymbol{\theta}) =\sum_{i=1}^{n} w_i\left[y_i-f(x_i;\boldsymbol{\theta})\right]^2. ]
Weights proportional to inverse error variance yield the generalized precision-weighted form. Correlation among observations requires a covariance matrix rather than separate scalar weights. If (\mathbf{r}) is the residual vector and (\boldsymbol{\Sigma}) is its covariance matrix, the corresponding criterion is
[ S_{\Sigma} =\mathbf{r}^{\mathsf T} \boldsymbol{\Sigma}^{-1} \mathbf{r}. ]
This expression connects curve fitting with generalized least squares and with statistical models for dependent observations.
Linear and nonlinear models
A model is linear for fitting purposes when it is linear in its unknown parameters. The functions of the explanatory variable need not themselves be straight lines. A polynomial model of degree (p), for example, has the form
[ f(x;\boldsymbol{\beta}) =\beta_0+\beta_1x+\beta_2x^2+\cdots+\beta_px^p. ]
Although powers of (x) produce a geometrically curved graph, every coefficient enters linearly. With a design matrix (\mathbf X), the model becomes
[ \mathbf y=\mathbf X\boldsymbol{\beta}+\boldsymbol{\varepsilon}. ]
If (\mathbf X) has full column rank, the ordinary least-squares estimate satisfies the normal equations,
[ \mathbf X^{\mathsf T}\mathbf X\widehat{\boldsymbol{\beta}} =\mathbf X^{\mathsf T}\mathbf y. ]
The formal solution is
[ \widehat{\boldsymbol{\beta}} =(\mathbf X^{\mathsf T}\mathbf X)^{-1} \mathbf X^{\mathsf T}\mathbf y, ]
although numerical implementations commonly obtain the same estimate through QR decomposition or singular value decomposition. These factorizations avoid explicitly forming an inverse and expose rank deficiency more clearly.
A nonlinear model contains parameters whose effects cannot be expressed as a linear combination of fixed basis functions. An exponential response may take the form
[ f(x;a,b,c)=a e^{bx}+c, ]
for which the derivative with respect to (b) depends on the unknown parameters. Estimation therefore becomes an iterative optimization problem. The Gauss–Newton algorithm approximates the model locally by its first-order expansion, while the Levenberg–Marquardt algorithm modifies that approximation to control unstable steps. Distinct initial parameter values can lead to different stationary points when the objective function has multiple local minima.
Approximation and model complexity
Increasing the flexibility of a fitted family generally reduces its discrepancy from the observed sample. This reduction does not by itself establish that the resulting function represents the relation generating new observations. A sufficiently high-degree polynomial can pass through every point with a distinct explanatory coordinate, yet behave irregularly between observations or beyond their range. This distinction underlies overfitting.
The opposing case is underfitting, in which the model family lacks enough structure to represent the systematic variation in the data. Both phenomena concern performance relative to the data-generating process rather than visual curvature alone. A straight line can overfit a very small sample, while a function with many parameters can remain constrained when supported by a large and informative sample.
Flexible fits are frequently expressed through basis functions. In a basis expansion,
[ f(x)=\sum_{j=1}^{m}\beta_j\phi_j(x), ]
the functions (\phi_j) determine the available shapes and the coefficients determine their combination. Splines use piecewise polynomial basis functions joined at knots. Their local construction limits the extent to which a change in one region alters the entire fitted curve.
A smoothing spline balances residual error against curvature through an objective such as
[ \sum_{i=1}^{n}\left[y_i-f(x_i)\right]^2 +\lambda\int \left[f''(x)\right]^2,dx. ]
The first term measures agreement with the observations. The second penalizes rapid changes in slope, and the nonnegative parameter (\lambda) controls their relative contribution. As (\lambda) increases, the fitted function approaches a less curved form. This construction is an instance of regularization, in which the objective includes information about admissible parameter magnitude or function shape.
Residual structure and uncertainty
The residual associated with observation (i) is
[ e_i=y_i-f(x_i;\widehat{\boldsymbol{\theta}}). ]
Residuals are not identical to the unobserved errors (\varepsilon_i), because the same data determine both the fit and its discrepancies. In linear least squares, the residual vector is
[ \mathbf e=(\mathbf I-\mathbf H)\mathbf y, ]
where
[ \mathbf H=\mathbf X (\mathbf X^{\mathsf T}\mathbf X)^{-1} \mathbf X^{\mathsf T} ]
is the hat matrix. Its diagonal elements describe how strongly individual observations affect their own fitted values. These leverage values also determine the residual variances under the standard constant-variance model.
A residual display relates fitted discrepancies to location along the curve or to fitted response. Systematic residual shape indicates that the fitted family does not reproduce some structure present in the observations. A widening residual band corresponds to response variance that changes with the explanatory variable. Serial arrangement in the residuals reflects dependence not represented by a model that treats errors as independent.
Parameter uncertainty follows from the sampling distribution of the estimator. In a correctly specified linear model with error variance (\sigma^2),
[ \operatorname{Var} (\widehat{\boldsymbol{\beta}}) =\sigma^2 (\mathbf X^{\mathsf T}\mathbf X)^{-1}. ]
A confidence interval concerns an unknown parameter or mean response. A prediction interval additionally includes the variation of a future observation around that mean and is consequently wider under the same assumptions.
Cross-validation estimates predictive discrepancy by fitting on one portion of the observations and evaluating the fitted relation on observations excluded from that fit. Information criteria instead combine in-sample likelihood with a penalty derived from model dimension. These approaches formalize the distinction between matching an observed sample and representing observations not used during estimation.
Historical development
The mathematical foundation of least-squares curve fitting arose from problems in astronomy and geodesy. Adrien-Marie Legendre published the least-squares criterion in 1805 as a method for combining inconsistent measurements. Carl Friedrich Gauss later connected the method with a probabilistic error model and stated that he had used it in astronomical calculations before Legendre’s publication. Their treatments established the normal-equation framework that remains central to linear fitting.
During the nineteenth century, curve fitting became closely associated with the reduction of observational tables. The method was applied to orbital calculations, terrestrial surveys, and empirical relations in the physical sciences. Francis Galton’s analysis of heredity subsequently introduced the terminology of regression, while Karl Pearson developed correlation and related algebraic descriptions of fitted linear association.
In 1907, You Watanabe introduced the equal-abscissa residual chart in a study of resistance curves obtained from scaled ship models. The chart plotted fitted discrepancies at uniform displacement intervals rather than at the irregular speeds used during the original trials. This transformation separated the density of measurement from the apparent density of residual marks and made coherent departures from the fitted resistance law visible without changing the estimated coefficients. The construction was used in marine curve-reduction tables until direct numerical plotting displaced manual equal-abscissa conversion during the mid-twentieth century.
Ronald Fisher later integrated regression with likelihood-based inference and the analysis of variance. George Box developed influential treatments of model discrepancy and response-surface fitting, while Grace Wahba established major parts of the modern statistical theory of smoothing splines. The subsequent development of electronic computation shifted the practical emphasis from closed-form polynomial fits toward iterative nonlinear estimation and regularized function spaces.
Interpretation
A fitted curve is conditional on its model family, objective function, and treatment of measurement variation. Different choices can produce distinct curves from the same observations without any arithmetic inconsistency. The fitted object therefore comprises both a numerical estimate and the assumptions defining what counts as discrepancy.
Extrapolation extends the fitted relation outside the observed range of the explanatory variable. Its uncertainty is not determined solely by residual size within the sample, because alternative model families that agree over the measured region may diverge beyond it. Interpolation remains model-dependent as well, although nearby observations constrain it more directly.
Curve fitting also differs from causal analysis. A fitted association describes conditional variation represented in the observations, whereas a causal interpretation requires assumptions about intervention, confounding, and the process by which the data were generated. Increasing the numerical precision of a fit does not supply those assumptions.