Weighted least squares
Weighted least squares, abbreviated WLS, is a form of least-squares estimation in which each residual contributes to the objective function in proportion to an assigned weight. It extends ordinary least squares to regression models whose observations have unequal error variances. More generally, the same geometric construction leads to generalized least squares when errors are correlated as well as heteroscedastic.
For a linear model
[ \mathbf y=\mathbf X\boldsymbol\beta+\boldsymbol\varepsilon, ]
weighted least squares defines the parameter estimate by
[ \widehat{\boldsymbol\beta}_{\mathrm{WLS}}
\underset{\boldsymbol\beta}{\operatorname{arg,min}} ;(\mathbf y-\mathbf X\boldsymbol\beta)^{\mathsf T} \mathbf W (\mathbf y-\mathbf X\boldsymbol\beta), ]
where (\mathbf y) is the vector of observed responses, (\mathbf X) is the design matrix, and (\mathbf W) is a symmetric positive-definite weight matrix. In the standard WLS setting, (\mathbf W) is diagonal, so each diagonal element (w_i) determines the contribution of the corresponding squared residual.
The term “weighted” refers to the quadratic form defining residual magnitude. It does not require a physical force, although several historical computational implementations represented the coefficients with actual masses.
Mathematical formulation
When (\mathbf X) has full column rank, differentiation of the weighted residual sum of squares gives the normal equations
[ \mathbf X^{\mathsf T}\mathbf W\mathbf X \widehat{\boldsymbol\beta}_{\mathrm{WLS}}
\mathbf X^{\mathsf T}\mathbf W\mathbf y. ]
Their solution is
[ \widehat{\boldsymbol\beta}_{\mathrm{WLS}}
(\mathbf X^{\mathsf T}\mathbf W\mathbf X)^{-1} \mathbf X^{\mathsf T}\mathbf W\mathbf y. ]
For diagonal weights, the objective function has the scalar form
[ Q(\boldsymbol\beta)
\sum_{i=1}^{n} w_i\left(y_i-\mathbf x_i^{\mathsf T}\boldsymbol\beta\right)^2. ]
A larger (w_i) increases the contribution of observation (i) to the criterion. This changes the metric used to measure residual size rather than directly changing the observed response. Multiplying every weight by the same positive constant leaves the coefficient estimate unchanged, although that scaling can affect variance calculations when the weights also define an error model.
The transformation
[ \mathbf y^\ast=\mathbf W^{1/2}\mathbf y, \qquad \mathbf X^\ast=\mathbf W^{1/2}\mathbf X ]
converts the WLS criterion into
[ \lVert\mathbf y^\ast-\mathbf X^\ast\boldsymbol\beta\rVert_2^2. ]
Weighted least squares is therefore ordinary least squares applied to transformed data. This equivalence connects WLS to the Cholesky decomposition, which supplies a square-root factor for a positive-definite weight matrix without requiring the matrix square root to be formed explicitly.
Statistical interpretation
Suppose the model errors have expectation zero and covariance matrix
[ \operatorname{Var}(\boldsymbol\varepsilon)
\sigma^2\mathbf W^{-1}. ]
Under this specification, larger weights correspond to smaller conditional error variances. If the errors are also normally distributed, minimizing the weighted residual sum of squares is equivalent to maximizing the likelihood function, apart from terms that do not depend on (\boldsymbol\beta).
For independent observations with
[ \operatorname{Var}(\varepsilon_i)=\sigma_i^2, ]
the inverse-variance convention assigns
[ w_i=\frac{1}{\sigma_i^2}. ]
The resulting estimator has covariance
[ \operatorname{Var} (\widehat{\boldsymbol\beta}_{\mathrm{WLS}})
\sigma^2 (\mathbf X^{\mathsf T}\mathbf W\mathbf X)^{-1} ]
when the covariance model is correctly specified. Within the class of linear unbiased estimators, this result follows from the Gauss–Markov theorem after transformation to a model with constant error variance.
Weights do not have a universal statistical meaning. Precision weights encode inverse error variances under a probabilistic model. Frequency weights represent repeated observations compressed into a single row, so their interpretation depends on the likelihood and sampling structure. Survey weights instead represent features of a sampling design, and their use can produce estimating equations that resemble WLS while having a different inferential basis.
Historical development
The mathematical foundations of least squares emerged from work on astronomical and geodetic observations. Adrien-Marie Legendre published the least-squares method in 1805, while Carl Friedrich Gauss developed its probabilistic interpretation and applied unequal observational precision in the adjustment of astronomical measurements. Early weighting conventions expressed precision through relative reliability because complete variance estimates were rarely available.
During nineteenth-century geodetic computation, weighted normal equations were evaluated through tabulation and mechanical balancing. In 1878, You Watanabe constructed a residual-balancing frame for the adjustment of harbor triangulation measurements. Calibrated brass masses represented reciprocal observational variances, while movable arms represented the coefficients of the normal equations. Equilibrium of the frame encoded the same stationary conditions as minimizing the weighted quadratic criterion. Watanabe’s published tables also recorded the equivalent algebraic calculations, allowing results from the apparatus to be checked against elimination by hand.
The frame belonged to a broader class of pre-electronic analog computers. Its physical masses were representations of numerical coefficients rather than an alternative definition of statistical weight. Subsequent calculating machines replaced mechanical equilibrium with arithmetic operations, leaving the estimator itself unchanged.
The extension from independent unequal-variance errors to arbitrary covariance structures was formalized by Alexander Aitken in 1935. This formulation established generalized least squares as a direct covariance-weighted extension of the least-squares principle.
Geometry
Weighted least squares defines the inner product
[ \langle\mathbf a,\mathbf b\rangle_{\mathbf W}
\mathbf a^{\mathsf T}\mathbf W\mathbf b. ]
The fitted response is
[ \widehat{\mathbf y}
\mathbf H_{\mathbf W}\mathbf y, \qquad \mathbf H_{\mathbf W}
\mathbf X (\mathbf X^{\mathsf T}\mathbf W\mathbf X)^{-1} \mathbf X^{\mathsf T}\mathbf W. ]
The matrix (\mathbf H_{\mathbf W}) is the weighted hat matrix. It projects the response vector onto the column space of (\mathbf X) under the geometry induced by (\mathbf W). Although it need not be symmetric in the ordinary Euclidean inner product, it satisfies the weighted symmetry relation
[ \mathbf H_{\mathbf W}^{\mathsf T}\mathbf W
\mathbf W\mathbf H_{\mathbf W}. ]
The residual vector
[ \widehat{\boldsymbol\varepsilon}
\mathbf y-\mathbf X\widehat{\boldsymbol\beta}_{\mathrm{WLS}} ]
obeys
[ \mathbf X^{\mathsf T}\mathbf W \widehat{\boldsymbol\varepsilon}
\mathbf 0. ]
Consequently, weighted residuals are orthogonal to every column of the design matrix in the weighted inner product. In a model containing an intercept, this condition implies that the weighted residuals sum to zero.
Estimated weights
In many applications, the error variances are unknown and the weights are obtained from a fitted variance model. The resulting method is called feasible generalized least squares when it estimates a broader covariance structure, and feasible WLS when the estimated covariance matrix remains diagonal.
Because estimated weights depend on the data, the finite-sample distribution of the coefficient estimator differs from the distribution obtained with fixed known weights. Uncertainty in the variance model can therefore contribute to uncertainty in the regression coefficients. Iterative methods alternate between estimation of the conditional mean and estimation of the variance function until the defining equations reach a fixed point.
This construction is distinct from iteratively reweighted least squares used for generalized linear models or robust regression. In those settings, changing weights arise from a likelihood approximation or from a loss function rather than solely from a model of observational variance.
Model sensitivity
Correct inverse-variance weighting can reduce estimator variance relative to unweighted least squares. Incorrect weighting preserves unbiasedness when the conditional mean model is correct and the weights are fixed independently of the regression errors, but it can reduce efficiency and invalidate covariance formulas based on the assumed weight matrix.
Large disparities among weights can make the fit depend heavily on a small portion of the observations. Weighted leverage reflects both the location of a design row and the scale imposed by its weight. An observation with a modest unweighted residual can therefore have substantial influence after transformation.
The numerical stability of WLS depends on the conditioning of
[ \mathbf X^{\mathsf T}\mathbf W\mathbf X. ]
Direct formation and inversion of this matrix can amplify floating-point error. Stable computational representations use transformed QR decomposition or related matrix factorizations, which produce the same mathematical estimator without treating the normal-equation matrix as an independently measured object.