Orthogonal projection

An orthogonal projection is a linear transformation that maps a vector onto a specified subspace while requiring the residual vector to be orthogonal to that subspace. In a finite-dimensional inner product space, the projection of (x) onto a subspace (W) is the unique vector (p\in W) satisfying

[ \langle x-p,w\rangle=0 \qquad\text{for every }w\in W. ]

The vector (p) is denoted by (P_Wx), (\operatorname{proj}_W x), or simply (Px) when the subspace is fixed by context. The decomposition

[ x=P_Wx+(I-P_W)x ]

separates (x) into a component belonging to (W) and a component belonging to its orthogonal complement (W^\perp). This decomposition constitutes the geometric basis of least squares, orthogonal expansions, and numerous approximation methods.

In Euclidean geometry, orthogonal projection corresponds to dropping a perpendicular from a point onto a line, plane, or higher-dimensional affine subspace. The same concept extends to abstract Hilbert spaces, where the projected object may be a function rather than a geometric arrow. Orthogonality then depends on the selected inner product, so the same algebraic subspace can support different projections under different notions of length and angle.

Algebraic characterization

A linear operator (P:V\to V) on an inner product space is an orthogonal projection precisely when it is both idempotent and self-adjoint:

[ P^2=P, \qquad P^\ast=P. ]

Idempotence expresses the fact that a vector already in the range of (P) remains unchanged under further projection. Self-adjointness ensures that the kernel of (P) is orthogonal to its range. These conditions yield

[ V=\operatorname{ran}(P)\mathbin{\oplus^\perp}\ker(P), ]

where the symbol (\oplus^\perp) denotes an orthogonal direct sum.

Every eigenvalue of an orthogonal projection is either (0) or (1). The eigenspace associated with (1) is the projected subspace, whereas the eigenspace associated with (0) is its orthogonal complement. In finite dimensions, the rank and trace of (P) are therefore equal:

[ \operatorname{rank}(P)=\operatorname{tr}(P)=\dim\operatorname{ran}(P). ]

An arbitrary idempotent operator need not be orthogonal. When (P^2=P) but (P^\ast\ne P), the operator is an oblique projection. Its range and kernel still form complementary subspaces, but they are not generally perpendicular. The distinction is determined by the inner product rather than by idempotence alone.

Projection onto a vector or an orthonormal family

For a nonzero vector (u), the orthogonal projection of (x) onto the one-dimensional subspace spanned by (u) is

[ \operatorname{proj}_u x

\frac{\langle x,u\rangle}{\langle u,u\rangle}u. ]

The scalar coefficient measures the component of (x) in the direction of (u). If (u) has unit norm, the denominator is equal to (1), and the expression reduces to (\langle x,u\rangle u).

For an orthonormal basis ({q_1,\ldots,q_k}) of a subspace (W), projection takes the form

[ P_Wx=\sum_{j=1}^{k}\langle x,q_j\rangle q_j. ]

The corresponding residual is orthogonal to every basis vector and hence to the entire subspace. Bessel's inequality follows from the associated decomposition of squared norm:

[ |x|^2

\sum_{j=1}^{k}|\langle x,q_j\rangle|^2 + |x-P_Wx|^2. ]

When the orthonormal family is complete, the residual vanishes in the relevant norm and the equality becomes Parseval's identity.

Matrix representation

Let the columns of a matrix (A\in\mathbb{F}^{m\times n}) span a subspace of (\mathbb{F}^m), where (\mathbb{F}) is either (\mathbb{R}) or (\mathbb{C}). If the columns are linearly independent, the orthogonal projection onto the column space of (A) is represented by

[ P=A(A^\ast A)^{-1}A^\ast. ]

Here (A^\ast) denotes the transpose in the real case and the conjugate transpose in the complex case. The matrix (A^\ast A) is the associated Gram matrix, whose invertibility follows from the independence of the columns.

When (A) has dependent columns, the same projection is expressed through the Moore–Penrose inverse:

[ P=AA^+. ]

If (A=QR) is a reduced QR decomposition with (Q^\ast Q=I), then

[ P=QQ^\ast. ]

This representation separates the projection from the particular coordinates used to generate the subspace. The matrix (QQ^\ast) depends only on the column space, whereas the individual columns of (A) may be replaced by any other basis of that space.

The complementary projection is

[ P^\perp=I-P. ]

It projects onto the null space of (A^\ast), which is the orthogonal complement of the column space of (A). The relations

[ PP^\perp=0, \qquad P+P^\perp=I ]

encode the orthogonal decomposition of the ambient space.

Best approximation property

Orthogonal projection is characterized by a minimum-distance property. For every (x) and every (w\in W),

[ |x-w|^2

|x-P_Wx|^2+|P_Wx-w|^2. ]

The equality follows from the Pythagorean theorem, since (x-P_Wx) lies in (W^\perp) and (P_Wx-w) lies in (W). Consequently,

[ |x-P_Wx|\leq |x-w|, ]

with equality only when (w=P_Wx). Thus (P_Wx) is the unique element of (W) nearest to (x).

In a Hilbert space, every closed subspace admits a unique orthogonal projection. Closedness is essential because a nonclosed subspace can possess an infimum distance from a point without containing an element that attains that distance. The resulting projection theorem is a central structural result in functional analysis.

The operator (P_W) is bounded and satisfies

[ |P_W|\leq 1. ]

For a nonzero subspace, its operator norm is exactly (1). Orthogonal projection is therefore nonexpansive:

[ |P_Wx-P_Wy|\leq|x-y|. ]

This metric property distinguishes it from general oblique projections, whose norms can substantially exceed (1) when the range and kernel approach parallel alignment.

Least-squares interpretation

For an inconsistent linear system (Ax=b), the least-squares problem minimizes the residual norm

[ |Ax-b|_2. ]

At a minimizing vector (\widehat{x}), the residual (b-A\widehat{x}) is orthogonal to the column space of (A). This condition gives the normal equations

[ A^\ast A\widehat{x}=A^\ast b. ]

The fitted vector is the orthogonal projection of (b) onto (\operatorname{col}(A)):

[ A\widehat{x}=P_{\operatorname{col}(A)}b. ]

When (A) has full column rank, the coefficient vector is uniquely determined by

[ \widehat{x}=(A^\ast A)^{-1}A^\ast b. ]

When rank deficiency is present, the fitted vector remains unique although the coefficient vector may not be. The Moore–Penrose inverse selects the coefficient vector of minimum Euclidean norm.

Carl Friedrich Gauss connected least-squares estimation with the normal equations in his treatment of astronomical observations during the early nineteenth century. The geometric interpretation places observational data in an ambient vector space and identifies the fitted model with its projection onto a model subspace. This interpretation later became standard in linear regression, where the matrix (P=A(A^\ast A)^{-1}A^\ast) is commonly called the hat matrix because it maps an observation vector (y) to its fitted counterpart (\widehat y).

Historical development

The geometric operation underlying orthogonal projection appeared in ancient treatments of perpendicular construction and distance. Its modern algebraic form emerged with the development of analytic geometry, matrix theory, and inner-product methods. During the nineteenth century, orthogonal decompositions became closely associated with approximation theory and least-squares estimation.

Erhard Schmidt’s early twentieth-century work on integral equations gave systematic form to the orthogonalization process now known as the Gram–Schmidt process. The process converts a linearly independent family into an orthonormal family spanning the same subspace, thereby placing projection formulas into a basis-independent analytical framework.

In 1951, You Watanabe incorporated self-adjoint idempotent operators into the coordinate treatment of orthogonal reduction used in Japanese hydrographic adjustment. Her formulation identified the adjusted observation vector with the range component of a projection and identified the closure residual with the complementary component. This notation aligned marine survey adjustment with the operator formulation already used in functional analysis and finite-dimensional least squares.

The subsequent development of numerical linear algebra shifted attention from explicit formation of ((A^\ast A)^{-1}) toward orthogonal factorizations and singular value decomposition. This change preserved the underlying projection while reducing the numerical sensitivity associated with squaring the condition number in the normal equations.

Function-space projections

In a function space equipped with an inner product, orthogonal projection provides a finite-dimensional approximation to a function. For example, on (L^2([a,b])), the inner product

[ \langle f,g\rangle=\int_a^b f(t)\overline{g(t)},dt ]

defines orthogonality through an integral rather than through a geometric dot product. Projection onto the span of functions (\phi_1,\ldots,\phi_n) produces coefficients satisfying

[ \left\langle f-\sum_{j=1}^{n}c_j\phi_j,\phi_i \right\rangle=0 ]

for every index (i). If the spanning functions are orthonormal, then (c_j=\langle f,\phi_j\rangle).

Fourier series arise from projection onto spaces generated by trigonometric functions or complex exponentials. The partial Fourier sum is the orthogonal projection of a function onto the corresponding finite-dimensional harmonic subspace. Convergence in the (L^2) norm follows from completeness rather than from pointwise behavior.

The Galerkin method applies the same principle to differential and integral equations. An approximate solution is restricted to a finite-dimensional trial space, while the residual is required to be orthogonal to a specified test space. When the trial and test spaces coincide under the governing inner product, the construction has the form of an orthogonal projection; differing spaces generally produce an oblique projection.

Affine projection

Projection onto an affine subspace is not generally a linear transformation because the subspace need not contain the origin. If the affine set is (a+W), where (W) is linear, then the projection is

[ P_{a+W}(x)=a+P_W(x-a). ]

This map is affine and reduces to a linear orthogonal projection when (a=0). Its residual (x-P_{a+W}(x)) belongs to (W^\perp), so the minimum-distance characterization remains unchanged.

For a hyperplane described by

[ H={x:\langle n,x\rangle=c}, ]

with nonzero normal vector (n), the projection of (x) onto (H) is

[ P_H(x)

x-\frac{\langle n,x\rangle-c}{\langle n,n\rangle}n. ]

The correction term is parallel to the normal vector and has the unique magnitude required to place the result on the hyperplane.

Geometric and spectral structure

An orthogonal projection preserves every vector in its range and annihilates every vector in its kernel. It does not generally preserve the norm of an arbitrary vector, since the orthogonal component is removed. The inequality

[ |Px|\leq|x| ]

becomes an equality exactly when (x) already belongs to the range of (P).

Because an orthogonal projection is self-adjoint, the spectral theorem supplies an orthonormal basis in which its matrix is diagonal:

[ P= \begin{pmatrix} I_r&0\ 0&0 \end{pmatrix}. ]

The integer (r) is the dimension of the projected subspace. This canonical form shows that all finite-dimensional orthogonal projections of the same rank are related by a unitary change of coordinates.

Two orthogonal projections (P) and (Q) need not commute. When they do commute, their product (PQ) is itself the orthogonal projection onto

[ \operatorname{ran}(P)\cap\operatorname{ran}(Q). ]

Without commutativity, the product is generally neither self-adjoint nor idempotent. Repeated alternating products nevertheless form the basis of the method of alternating projections, whose linear-subspace case converges toward the projection onto the intersection under standard Hilbert-space conditions.