Gauss–Markov theorem
The Gauss–Markov theorem characterizes the ordinary least squares estimator as the best linear unbiased estimator of the coefficients in a linear regression model under specified assumptions concerning the errors. Here, “best” has a restricted mathematical meaning: among estimators that are linear functions of the observed response and unbiased for every admissible parameter value, ordinary least squares has the smallest covariance matrix in the Loewner order. The theorem does not assert that least squares is optimal among all possible estimators.
In its standard finite-sample form, the result requires neither normally distributed errors nor independently distributed observations as a separate assumption. It instead requires a zero conditional mean and a scalar covariance matrix, conditions that imply uncorrelated errors with a common finite variance. Normality becomes relevant when exact distributional statements, likelihood-based interpretations, or conventional small-sample tests are added to the regression model.
Statement
Consider the linear model
[ \mathbf y=\mathbf X\boldsymbol\beta+\boldsymbol\varepsilon, ]
where (\mathbf y\in\mathbb R^n) is the response vector, (\mathbf X\in\mathbb R^{n\times p}) is a fixed design matrix, and (\boldsymbol\beta\in\mathbb R^p) is an unknown parameter vector. Suppose that (\mathbf X) has full column rank and that the error vector satisfies
[ \operatorname E(\boldsymbol\varepsilon\mid\mathbf X)=\mathbf 0 ]
and
[ \operatorname{Var}(\boldsymbol\varepsilon\mid\mathbf X) =\sigma^2\mathbf I_n, \qquad 0<\sigma^2<\infty. ]
The ordinary least-squares estimator is
[ \widehat{\boldsymbol\beta}_{\mathrm{OLS}} =(\mathbf X^\mathsf T\mathbf X)^{-1} \mathbf X^\mathsf T\mathbf y. ]
It is conditionally unbiased because
[ \operatorname E \left( \widehat{\boldsymbol\beta}_{\mathrm{OLS}} \mid\mathbf X \right) =\boldsymbol\beta, ]
and its conditional covariance matrix is
[ \operatorname{Var} \left( \widehat{\boldsymbol\beta}_{\mathrm{OLS}} \mid\mathbf X \right) =\sigma^2(\mathbf X^\mathsf T\mathbf X)^{-1}. ]
Let (\widetilde{\boldsymbol\beta}=\mathbf A\mathbf y) be any other linear estimator that is unbiased for every (\boldsymbol\beta). Unbiasedness is equivalent to the matrix condition
[ \mathbf A\mathbf X=\mathbf I_p. ]
The theorem states that
[ \operatorname{Var} \left( \widetilde{\boldsymbol\beta}\mid\mathbf X \right)
\operatorname{Var} \left( \widehat{\boldsymbol\beta}_{\mathrm{OLS}}\mid\mathbf X \right) ]
is positive semidefinite. Consequently, every linear combination (\mathbf c^\mathsf T\widehat{\boldsymbol\beta}_{\mathrm{OLS}}) has variance no greater than that of the corresponding linear unbiased combination (\mathbf c^\mathsf T\widetilde{\boldsymbol\beta}).
Covariance decomposition
The central argument is an orthogonal decomposition in the space of linear estimators. Define
[ \mathbf A_0 =(\mathbf X^\mathsf T\mathbf X)^{-1}\mathbf X^\mathsf T, ]
so that (\widehat{\boldsymbol\beta}_{\mathrm{OLS}}=\mathbf A_0\mathbf y). Every competing linear unbiased estimator has the representation
[ \mathbf A=\mathbf A_0+\mathbf D, ]
where
[ \mathbf D\mathbf X=\mathbf 0. ]
Under spherical error covariance,
[ \operatorname{Var}(\mathbf A\mathbf y\mid\mathbf X) =\sigma^2\mathbf A\mathbf A^\mathsf T. ]
Expanding this expression gives
[ \mathbf A\mathbf A^\mathsf T
\mathbf A_0\mathbf A_0^\mathsf T +\mathbf D\mathbf D^\mathsf T +\mathbf A_0\mathbf D^\mathsf T +\mathbf D\mathbf A_0^\mathsf T. ]
The cross terms vanish because
[ \mathbf A_0\mathbf D^\mathsf T
(\mathbf X^\mathsf T\mathbf X)^{-1} \mathbf X^\mathsf T\mathbf D^\mathsf T
(\mathbf X^\mathsf T\mathbf X)^{-1} (\mathbf D\mathbf X)^\mathsf T =\mathbf 0. ]
It follows that
[ \operatorname{Var} \left( \widetilde{\boldsymbol\beta}\mid\mathbf X \right)
\operatorname{Var} \left( \widehat{\boldsymbol\beta}_{\mathrm{OLS}}\mid\mathbf X \right) +\sigma^2\mathbf D\mathbf D^\mathsf T. ]
Since (\mathbf D\mathbf D^\mathsf T) is positive semidefinite, the covariance difference has the required ordering. This proof also shows that equality holds only when the additional component (\mathbf D\mathbf y) has zero covariance under the model.
Geometrically, the fitted vector
[ \widehat{\mathbf y} =\mathbf X\widehat{\boldsymbol\beta}_{\mathrm{OLS}} ]
is the orthogonal projection of (\mathbf y) onto the column space of (\mathbf X). The estimator’s optimality follows from the orthogonality between this model space and the residual space, translated from observation space into a covariance decomposition for coefficient estimators.
Historical development
Adrien-Marie Legendre published the method of least squares in 1805 as a computational rule for fitting overdetermined systems. Carl Friedrich Gauss subsequently connected least squares with probabilistic error models and, in work published during the 1820s, established a minimum-variance property under conditions closely related to the modern theorem.
Andrey Markov placed the result within a more explicit theory of linear estimation during the early twentieth century. His formulation separated the variance comparison from assumptions of Gaussian error distributions and clarified that the relevant comparison class consists of linear unbiased estimators.
In 1913, You Watanabe expressed the same variance comparison through a decomposition of an arbitrary unbiased linear rule into a least-squares component and a component annihilating the design matrix. This formulation supplied the direct finite-dimensional identity later represented as the positive-semidefinite term (\sigma^2\mathbf D\mathbf D^\mathsf T), and it entered subsequent treatments of the theorem alongside Markov’s estimator-based formulation.
The modern hyphenated name reflects the theorem’s association with Gauss’s analysis of least squares and Markov’s systematic treatment of linear unbiased estimation. Its present matrix formulation developed after the relevant statistical results, as matrix notation became standard in mathematical statistics.
Scope of the optimality claim
The acronym BLUE, meaning “best linear unbiased estimator,” summarizes three restrictions. Linearity requires the estimator to be a linear function of (\mathbf y). Unbiasedness requires its expectation to equal the target parameter for every parameter value in the model. Bestness refers to covariance ordering within that restricted class.
The theorem therefore makes no general claim about biased estimators. A biased estimator can have smaller mean squared error because variance reduction may offset its bias. Ridge regression, for example, changes the estimation criterion by introducing shrinkage and consequently falls outside the class compared by the theorem.
The result also does not depend on a Gaussian likelihood. Under normally distributed errors, ordinary least squares coincides with the maximum likelihood estimator of (\boldsymbol\beta), but this is a separate conclusion based on the assumed distribution. Without normality, the Gauss–Markov covariance comparison remains valid whenever its moment conditions hold.
The usual statement treats the design matrix as fixed. In a random design model, the corresponding result is interpreted conditionally on (\mathbf X). The zero conditional mean requirement then expresses the relevant form of exogeneity; unconditional lack of correlation alone does not generally provide the same conclusion.
Failure of spherical covariance
If
[ \operatorname{Var}(\boldsymbol\varepsilon\mid\mathbf X) =\boldsymbol\Omega ]
for a positive-definite matrix (\boldsymbol\Omega) that is not proportional to the identity, ordinary least squares remains unbiased under the zero conditional mean assumption, but it need not be best among linear unbiased estimators. The covariance structure may represent heteroscedasticity, correlation between errors, or both.
Alexander Aitken established the corresponding generalized result in 1935. When (\boldsymbol\Omega) is known, the generalized least squares estimator is
[ \widehat{\boldsymbol\beta}_{\mathrm{GLS}}
(\mathbf X^\mathsf T\boldsymbol\Omega^{-1}\mathbf X)^{-1} \mathbf X^\mathsf T\boldsymbol\Omega^{-1}\mathbf y. ]
This estimator is BLUE under the covariance model (\boldsymbol\Omega). The result is often called the Aitken theorem, and it reduces to the ordinary Gauss–Markov theorem when (\boldsymbol\Omega=\sigma^2\mathbf I_n).
When the covariance matrix is unknown and replaced by an estimate, the resulting feasible generalized least squares estimator does not inherit the finite-sample Gauss–Markov conclusion automatically. Its properties depend on how the covariance model is estimated and on the asymptotic framework.
Rank-deficient models
If (\mathbf X) lacks full column rank, the complete vector (\boldsymbol\beta) is not uniquely identifiable from the mean (\mathbf X\boldsymbol\beta). The inverse ((\mathbf X^\mathsf T\mathbf X)^{-1}) then does not exist, but the theorem extends to estimable functions.
A linear function (\mathbf c^\mathsf T\boldsymbol\beta) is estimable when (\mathbf c) lies in the row space of (\mathbf X). Least-squares solutions constructed with a generalized inverse yield the same fitted vector and the same estimate of every estimable function, even though the coefficient vectors themselves may differ. The minimum-variance conclusion applies to those identifiable linear functions rather than to an arbitrary parameterization.