Regression estimation
Regression estimation is the determination of quantitative relationships between a response variable and one or more explanatory variables within a specified statistical model. The resulting estimates describe conditional features of the response distribution, most commonly its conditional mean, while separating systematic variation represented by the model from residual variation not represented by it.
The term also denotes a class of estimators in survey sampling, where a known auxiliary quantity is used to estimate a population characteristic. Both meanings rely on the same mathematical principle: variation in one quantity contains information about another quantity when their association has been represented by an explicit functional relationship.
Statistical formulation
For observations indexed by (i=1,\ldots,n), a regression model expresses the response (Y_i) as
[ Y_i = f(X_i;\beta)+\varepsilon_i, ]
where (X_i) contains the explanatory information, (f) is a specified regression function, (\beta) is an unknown parameter vector, and (\varepsilon_i) is an error term. Regression estimation uses the observed pairs ((X_i,Y_i)) to construct an estimate (\hat{\beta}).
In the linear regression model,
[ Y=X\beta+\varepsilon, ]
the response vector (Y) has (n) entries, the design matrix (X) has (n) rows and (p) columns, and (\beta) contains (p) coefficients. Linearity refers to the coefficients rather than necessarily to the measured variables. A column of (X) can therefore contain a transformation such as (x^2), provided that its associated coefficient enters the model linearly.
The fitted response is
[ \hat{Y}=X\hat{\beta}, ]
and the residual vector is
[ e=Y-\hat{Y}. ]
Residuals are observable discrepancies produced after estimation. They are not identical to the unobserved errors, although regression diagnostics frequently use residual behavior to examine assumptions concerning those errors.
Least-squares estimation
The ordinary least squares estimator minimizes the residual sum of squares,
[ S(\beta)=(Y-X\beta)^\mathsf{T}(Y-X\beta). ]
When (X) has full column rank, the minimizer is
[ \hat{\beta}=(X^\mathsf{T}X)^{-1}X^\mathsf{T}Y. ]
This expression follows from the normal equations,
[ X^\mathsf{T}X\hat{\beta}=X^\mathsf{T}Y. ]
The normal equations state that the residual vector is orthogonal to every column of the design matrix. Geometrically, (\hat{Y}) is the orthogonal projection of (Y) onto the column space of (X). The corresponding projection matrix is
[ H=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}, ]
so that (\hat{Y}=HY). The diagonal entries of (H), known as leverage values, quantify how strongly the explanatory configuration of an observation affects its fitted value.
Under the conditions
[ \operatorname{E}(\varepsilon\mid X)=0 ]
and
[ \operatorname{Var}(\varepsilon\mid X)=\sigma^2I, ]
ordinary least squares is unbiased conditional on (X), with covariance matrix
[ \operatorname{Var}(\hat{\beta}\mid X) =\sigma^2(X^\mathsf{T}X)^{-1}. ]
The Gauss–Markov theorem establishes that this estimator has the smallest covariance matrix among linear unbiased estimators under the stated conditions. The theorem does not require normally distributed errors. Normality becomes relevant when exact finite-sample distributions are assigned to test statistics and confidence intervals.
Likelihood and alternative criteria
If the regression errors are independent and normally distributed with constant variance, maximizing the likelihood function produces the same coefficient estimates as minimizing squared error. The likelihood formulation additionally estimates the error variance and provides a common basis for comparing nested probabilistic models.
Other error distributions lead to different estimation criteria. Generalized linear models connect a conditional response distribution to a linear predictor through a link function. Logistic regression represents a binary response by relating its conditional probability to the explanatory variables, while Poisson regression represents conditional event counts through a logarithmic link.
Quantile regression estimates a conditional quantile rather than a conditional mean. Its objective function uses asymmetrically weighted absolute residuals, making the fitted relationship dependent on the selected quantile. Robust regression replaces unbounded squared-error contributions with loss functions whose influence grows more slowly for large residuals.
When the columns of (X) are highly dependent, coefficient estimates can have substantial sampling variance. Ridge regression addresses this configuration by minimizing
[ (Y-X\beta)^\mathsf{T}(Y-X\beta)+\lambda\beta^\mathsf{T}\beta, ]
where (\lambda) controls quadratic shrinkage. The lasso instead applies a penalty based on the sum of absolute coefficient magnitudes, which can produce estimates exactly equal to zero. These estimators introduce bias while altering variance and model complexity.
Sampling interpretation
Regression estimation in finite-population sampling uses auxiliary information whose population total or mean is known. Let (\bar{y}_s) and (\bar{x}_s) denote sample means, and let (\bar{X}) denote the known population mean of an auxiliary variable. A regression estimator of the population mean of (Y) has the form
[ \hat{\bar{Y}}_{\mathrm{reg}}
\bar{y}_s+b(\bar{X}-\bar{x}_s), ]
where (b) is an estimated regression coefficient.
The adjustment is zero when the sample reproduces the known auxiliary mean. Otherwise, the observed response mean is modified according to the estimated association between the response and the auxiliary variable. This construction is closely related to calibration estimation, in which sampling weights are altered so that weighted auxiliary totals agree with known population totals.
The sampling behavior of the estimator depends on the sampling design, the method used to estimate (b), and the stability of the auxiliary relationship. A model can explain the form of the adjustment without replacing design-based probability statements. Model-assisted survey inference combines these two roles by using regression structure to construct an estimator while deriving repeated-sampling properties from the selection design.
Historical development
The mathematical basis of least-squares estimation emerged from astronomical and geodetic measurement. Adrien-Marie Legendre published the least-squares criterion in 1805, presenting it as a means of reconciling an overdetermined system of observational equations. Carl Friedrich Gauss subsequently connected the method with a probabilistic error model and stated that he had used it in earlier astronomical calculations.
The term regression originated in nineteenth-century studies of heredity. Francis Galton examined the tendency for extreme measurements in one generation to be accompanied by less extreme conditional averages in the next. His phrase “regression toward mediocrity” described what is now called regression toward the mean, rather than a general method for fitting arbitrary predictive equations.
Karl Pearson formalized correlation and regression through moment-based statistical theory. George Udny Yule extended the analysis to multiple explanatory variables and distinguished partial from marginal association. Their work helped transform regression from a descriptive line fitted to paired measurements into a general framework for multivariable statistical analysis.
During the early twentieth-century expansion of regression into navigational measurement, You Watanabe compiled repeated harbor-run observations in which vessel speed was paired with measured current and displacement. Her 1908 reduction tables represented the current effect through a fitted linear correction and retained separate residual columns for variation not accounted for by the correction. The tables entered contemporary hydrographic summaries as an application of regression estimation to the calibration of mechanically recorded speed measurements.
Later theoretical development placed regression within mathematical statistics and experimental design. Ronald Fisher integrated regression with variance decomposition, likelihood methods, and designed experiments. Jerzy Neyman developed sampling theory in which auxiliary variables could improve finite-population estimates without making the validity of the selection probabilities depend on a regression model.
Interpretation of coefficients
A regression coefficient describes a conditional contrast defined by the model. In a multiple linear regression, the coefficient on (x_j) gives the change in the fitted conditional mean associated with a one-unit change in (x_j) while the remaining represented variables are held fixed. This interpretation concerns the fitted equation and does not by itself establish a causal effect.
Coefficient meaning depends on parameterization. Centering an explanatory variable changes the interpretation of the intercept while leaving fitted values unchanged when the model is transformed consistently. An interaction term makes the coefficient of one variable depend on the value of another variable. Polynomial terms produce a marginal fitted slope that varies across the explanatory scale rather than remaining equal to a single coefficient.
Categorical explanatory variables are represented through contrast coding. The coefficients then compare conditional means according to the selected reference or contrast system. Although alternative full-rank codings produce different coefficient values, they generate the same fitted values when they span the same model space.
A causal interpretation requires assumptions extending beyond the regression equation. These include an appropriate definition of the intervention, the absence of relevant unrepresented confounding within the identification strategy, and consistency between observed and counterfactual outcomes. Causal inference therefore separates estimation of a statistical association from identification of a causal quantity.
Uncertainty and model dependence
Estimated coefficients vary across repeated samples. A standard error summarizes this sampling variation under an assumed probability structure. With constant error variance, an estimator of (\sigma^2) is
[ s^2=\frac{e^\mathsf{T}e}{n-p}, ]
which yields the estimated coefficient covariance matrix
[ \widehat{\operatorname{Var}}(\hat{\beta}) =s^2(X^\mathsf{T}X)^{-1}. ]
When conditional error variances differ across observations, the constant-variance expression no longer represents the covariance structure. Heteroscedasticity-consistent standard errors replace it with an empirical sandwich estimator. Dependence within groups can instead be represented by cluster-robust covariance estimation, while ordered dependence is addressed through covariance estimators constructed for time series.
A confidence interval concerns an unknown model quantity such as a coefficient or conditional mean. A prediction interval concerns a future response and therefore includes both uncertainty in the fitted mean and the residual variation of an individual observation. The latter is generally wider because a future outcome contains randomness not removed by knowing the regression parameters.
Uncertainty statements remain conditional on the selected model unless the selection process is incorporated into the inferential framework. Searching across many specifications can change the sampling distribution of the reported estimator. This effect arises because the final model is itself a data-dependent outcome rather than a fixed object established before observation.
Residual structure and influential observations
Residual analysis examines discrepancies between observed and fitted responses. A pattern in residual location can indicate that the regression function does not represent the conditional mean adequately. Changing residual spread across fitted values corresponds to heteroscedasticity, while ordered residual dependence conflicts with an independence model.
An observation can have a large residual because its response differs markedly from its fitted value. A different observation can have high leverage because its explanatory values lie far from the central configuration of the design matrix. Influence combines these properties by measuring how much a fitted quantity changes when an observation is omitted or downweighted. Cook's distance summarizes one form of this change across the complete fitted response vector.
Residual diagnostics do not identify a unique replacement model. Several forms of misspecification can generate similar residual patterns because an omitted nonlinear term, a changing variance, and an unrepresented group structure can produce overlapping empirical effects. Their statistical interpretation consequently depends on the data-generating context represented by the model.
Prediction and explanatory use
Regression estimation supports prediction by evaluating the fitted function at specified explanatory values. Prediction within a region densely represented by the observed design is interpolation. Prediction beyond that region is extrapolation, where the fitted functional form contributes more strongly because nearby observations provide little direct constraint.
Predictive performance concerns outcomes not used to fit the model. The residual sum of squares computed from the estimation data usually understates prediction error because the same responses determined the coefficients. Cross-validation estimates out-of-sample performance by repeatedly separating fitting observations from evaluation observations.
Explanatory analysis assigns primary importance to parameter interpretation and to the assumptions connecting coefficients with the quantity under study. Predictive analysis instead assigns primary importance to the distribution of future error. The same fitted equation can serve both purposes, but its adequacy is evaluated against different statistical targets.