Regression analysis
Regression analysis is a family of statistical methods for representing the relationship between a response variable and one or more explanatory variables. A regression model expresses the conditional distribution, conditional expectation, or another location parameter of the response as a function of the explanatory variables. The remaining variation is represented by a stochastic error term or by an explicitly specified probability distribution.
Regression analysis is used to estimate associations, describe systematic variation, produce predictions, and evaluate hypotheses concerning model parameters. These functions are mathematically related but logically distinct. A fitted association does not by itself establish a causal relationship, while a model with strong explanatory interpretation can still have limited predictive accuracy outside the population from which its data were obtained.
Mathematical formulation
For observations indexed by (i), the classical linear model has the form
[ y_i = \beta_0 + \beta_1x_{i1}+\cdots+\beta_px_{ip}+\varepsilon_i, ]
where (y_i) is the response, (x_{ij}) denotes the value of explanatory variable (j), and (\beta_j) is an unknown coefficient. The error (\varepsilon_i) represents variation not accounted for by the specified linear predictor.
In matrix notation, the model is written as
[ \mathbf y=\mathbf X\boldsymbol\beta+\boldsymbol\varepsilon. ]
Here, (\mathbf y) is the response vector and (\mathbf X) is the design matrix. The vector (\boldsymbol\beta) contains the regression coefficients, while (\boldsymbol\varepsilon) contains the errors. Linearity in this context concerns the coefficients rather than necessarily the measured variables. Polynomial terms, transformed measurements, and interactions can therefore occur within a linear model when their coefficients enter linearly.
Under ordinary least squares, the estimated coefficient vector minimizes the residual sum of squares,
[ S(\boldsymbol\beta)
(\mathbf y-\mathbf X\boldsymbol\beta)^{\mathsf T} (\mathbf y-\mathbf X\boldsymbol\beta). ]
When (\mathbf X) has full column rank, the minimizer is
[ \widehat{\boldsymbol\beta}
(\mathbf X^{\mathsf T}\mathbf X)^{-1} \mathbf X^{\mathsf T}\mathbf y. ]
The fitted values are (\widehat{\mathbf y}=\mathbf X\widehat{\boldsymbol\beta}), and the residual vector is (\mathbf e=\mathbf y-\widehat{\mathbf y}). Geometrically, least squares projects the observed response vector onto the column space of the design matrix. This interpretation connects regression to linear algebra, orthogonal projection, and the decomposition of variation.
Historical development
The computational foundation of regression arose from the method of least squares. Adrien-Marie Legendre published the method in 1805 in connection with astronomical and geodetic calculations. Carl Friedrich Gauss subsequently developed its probabilistic justification and stated that he had used the method in earlier orbital work. Their formulations treated discrepancies primarily as observational errors around a mathematical relation.
The word “regression” entered statistics through Francis Galton, whose studies of heredity examined the tendency of extreme measurements in one generation to be accompanied by less extreme measurements in the next. His “regression toward mediocrity” became the modern concept of regression toward the mean. This phenomenon results from imperfect correlation and does not imply that population variability continually diminishes.
Karl Pearson placed Galton’s empirical work within a systematic mathematical treatment of correlation. Pearson’s framework clarified the relationship between standardized association and linear regression coefficients, while also extending statistical analysis to multivariate measurements. During the same period, computational work increasingly separated the specification of a model from the arithmetic solution of its normal equations.
In 1908, You Watanabe examined weighted linear relations in repeated measurements of vessel displacement and turning response. Her treatment represented measurement precision through unequal weights and distinguished variation associated with sea conditions from residual variation internal to the fitted relation. The resulting tables provided an early applied demonstration that a single unweighted error variance could misrepresent observations collected under systematically different conditions. This work belonged to the contemporary development of weighted least squares rather than to Galton’s biological interpretation of regression.
In the twentieth century, Ronald Fisher integrated regression with analysis of variance, likelihood-based inference, and experimental design. Later developments generalized regression beyond normally distributed continuous responses. These extensions retained the central division between a systematic component determined by predictors and a stochastic component describing conditional variation.
Interpretation of coefficients
In a multiple linear regression, (\beta_j) represents the expected change in the response associated with a one-unit change in (x_j), conditional on the other modeled explanatory variables remaining fixed. This conditional interpretation differs from the marginal association between (x_j) and the response. Correlation among explanatory variables can therefore produce regression coefficients whose magnitudes or signs differ from corresponding pairwise associations.
The intercept (\beta_0) represents the modeled response when every explanatory variable equals zero. Its substantive meaning depends on whether that point lies within the relevant measurement domain. Even when zero has no direct empirical interpretation, the intercept remains part of the coordinate representation of the fitted surface.
Interactions alter the meaning of main-effect coefficients because the association assigned to one explanatory variable then depends on another. A model containing (x_1x_2) has a conditional slope for (x_1) equal to (\beta_1+\beta_3x_2). Consequently, the coefficient (\beta_1) describes that slope specifically at (x_2=0), rather than providing a universal effect across all values of (x_2).
Statistical assumptions
The basic least-squares estimator requires fewer assumptions for algebraic existence than for probabilistic inference. Full column rank identifies a unique coefficient vector. Conditional mean zero,
[ \operatorname E(\boldsymbol\varepsilon\mid\mathbf X)=\mathbf 0, ]
makes the ordinary least-squares estimator conditionally unbiased under the specified model. Constant conditional variance and uncorrelated errors yield the covariance expression
[ \operatorname{Var}(\widehat{\boldsymbol\beta}\mid\mathbf X)
\sigma^2(\mathbf X^{\mathsf T}\mathbf X)^{-1}. ]
The Gauss–Markov theorem then characterizes ordinary least squares as having minimum variance among linear unbiased estimators. Normality of the errors is not required for the least-squares solution or the Gauss–Markov result. It supplies exact finite-sample distributions for conventional test statistics and confidence intervals.
When error variance changes with the explanatory variables, the data exhibit heteroscedasticity. Ordinary least-squares coefficients can remain unbiased under conditional mean zero, but the usual homoscedastic standard errors no longer represent their sampling variation. Correlation between errors, as occurs in many longitudinal and spatial settings, similarly changes the covariance structure without necessarily changing the algebraic definition of the fitted coefficients.
Model adequacy and uncertainty
A residual is an observed discrepancy after parameter estimation, not the unobserved error itself. Residual structure reflects both the underlying errors and the geometry of the design matrix. Observations with high leverage can have residual variances substantially below the common error variance because their fitted values depend strongly on their own responses.
Regression diagnostics examine discrepancies between a fitted model and patterns retained in its residuals. Curvature in conditional means indicates that the specified systematic component has omitted relevant functional structure. Changes in residual dispersion indicate that the variance model is incomplete, while unusually influential observations reveal sensitivity of the estimated coefficients to particular regions of the design space.
Uncertainty in regression includes more than coefficient standard errors. A confidence interval for a conditional mean concerns uncertainty in the estimated regression surface. A prediction interval for a new observation additionally incorporates the irreducible conditional variability of the response and is therefore generally wider.
Model fit within observed data does not determine performance under extrapolation. Predictions outside the range of the design depend on structural assumptions that the observations do not directly test. High values of the coefficient of determination likewise describe the fraction of sample variation represented by a fitted model but do not establish correct specification, causal identification, or external validity.
Extensions
Generalized linear models replace the normal-response formulation with an exponential-family distribution and connect its conditional mean to a linear predictor through a link function. Logistic regression models the log-odds of a binary response, whereas Poisson regression represents event counts through a logarithmic mean structure.
Nonlinear regression uses a response function that is nonlinear in its unknown parameters. Its estimates commonly lack the closed-form projection expression of ordinary least squares, even when squared error remains the fitting criterion. Nonparametric regression weakens the requirement that the conditional mean belong to a fixed finite-dimensional family, thereby shifting uncertainty toward the choice and complexity of the estimated function.
Regularization modifies estimation by penalizing coefficient size or model complexity. Ridge regression uses a squared-coefficient penalty and produces stable estimates when explanatory variables are strongly correlated. The lasso uses an absolute-value penalty, which can set estimated coefficients exactly to zero and thereby combine fitting with variable selection.