Moore–Penrose inverse

The Moore–Penrose inverse is a uniquely defined generalization of the matrix inverse to rectangular and singular matrices. For a matrix (A), it is denoted by (A^{+}) and is also called the Moore–Penrose pseudoinverse. Unlike an ordinary inverse, it does not generally transform (A) into an identity matrix. Instead, its products with (A) are orthogonal projections associated with the row and column spaces of (A).

For every real or complex matrix, the Moore–Penrose inverse exists and is unique. It provides a canonical expression for least-squares solutions, minimum-norm solutions of underdetermined systems, and orthogonal projections onto linear subspaces. Its definition is invariant under unitary changes of coordinates and is closely connected with the singular value decomposition.

Definition

Let (A\in\mathbb{C}^{m\times n}). Its Moore–Penrose inverse is the unique matrix (A^{+}\in\mathbb{C}^{n\times m}) satisfying the four Penrose equations:

[ AA^{+}A=A, ]

[ A^{+}AA^{+}=A^{+}, ]

[ (AA^{+})^{*}=AA^{+}, ]

[ (A^{+}A)^{*}=A^{+}A, ]

where (^{*}) denotes the conjugate transpose. Over the real numbers, the conjugate transpose reduces to the ordinary transpose.

The first two identities make (A^{+}) a reflexive generalized inverse of (A). The remaining identities impose orthogonality conditions that distinguish the Moore–Penrose inverse from other generalized inverses. In particular, (AA^{+}) is the orthogonal projection onto the column space (\mathcal{R}(A)), while (A^{+}A) is the orthogonal projection onto the row space (\mathcal{R}(A^{*})).

If (A) is square and invertible, then

[ A^{+}=A^{-1}. ]

If (A=0), then (A^{+}=0). The involutive relation

[ (A^{+})^{+}=A ]

holds for every finite-dimensional matrix.

Construction from the singular value decomposition

Suppose that (A) has a singular value decomposition

[ A=U\Sigma V^{*}, ]

where (U) and (V) are unitary matrices and (\Sigma) is rectangular diagonal. If the nonzero singular values are

[ \sigma_1,\ldots,\sigma_r, ]

then (\Sigma^{+}) is formed by replacing every nonzero (\sigma_i) with (\sigma_i^{-1}), transposing the rectangular diagonal arrangement, and leaving every zero entry unchanged. The Moore–Penrose inverse is then

[ A^{+}=V\Sigma^{+}U^{*}. ]

This representation proves existence because the resulting matrix satisfies all four Penrose equations. It also exhibits uniqueness through the orthogonal decomposition of the domain and codomain into the range and null spaces of (A) and (A^{*}).

The singular value decomposition shows that the pseudoinverse acts as an ordinary reciprocal on directions associated with nonzero singular values. Directions belonging to the null space are mapped to zero. Consequently,

[ \mathcal{R}(A^{+})=\mathcal{R}(A^{*}) ]

and

[ \mathcal{N}(A^{+})=\mathcal{N}(A^{*}), ]

where (\mathcal{R}) denotes the range and (\mathcal{N}) denotes the null space.

Linear systems and least squares

For the system

[ Ax=b, ]

the vector

[ x_0=A^{+}b ]

has a canonical optimization interpretation. If the system is consistent, (x_0) is the unique exact solution having minimum Euclidean norm. If the system is inconsistent, (x_0) minimizes the residual norm

[ \lVert Ax-b\rVert_2 ]

and, among all vectors producing the minimum residual, has minimum Euclidean norm.

The fitted vector is

[ Ax_0=AA^{+}b, ]

which is the orthogonal projection of (b) onto (\mathcal{R}(A)). The residual

[ b-AA^{+}b ]

therefore lies in (\mathcal{N}(A^{*})), the orthogonal complement of the column space.

Every least-squares solution has the form

[ x=A^{+}b+(I-A^{+}A)z, ]

where (z\in\mathbb{C}^{n}). The factor (I-A^{+}A) is the orthogonal projection onto (\mathcal{N}(A)). The pseudoinverse solution corresponds to the component orthogonal to that null space.

When (A) has full column rank,

[ A^{+}=(A^{}A)^{-1}A^{}. ]

When (A) has full row rank,

[ A^{+}=A^{}(AA^{})^{-1}. ]

These formulas are special cases of the singular-value construction. They do not extend by direct inversion when the relevant Gram matrix is singular.

Algebraic properties

Scalar multiplication obeys

[ (\alpha A)^{+}=\alpha^{-1}A^{+} ]

for every nonzero scalar (\alpha). The adjoint operation commutes with pseudoinversion:

[ (A^{})^{+}=(A^{+})^{}. ]

For unitary matrices (U) and (V),

[ (UAV)^{+}=V^{}A^{+}U^{}. ]

This identity expresses the unitary invariance of the construction. It also accounts for the direct reduction of the problem to the diagonal singular-value matrix.

In general,

[ (AB)^{+}\neq B^{+}A^{+}. ]

The reverse-order identity holds under additional compatibility conditions involving the ranges of the factors and their adjoints. Its failure in the unrestricted case reflects the interaction between the intermediate range of (B) and the null space of (A).

For a normal matrix, pseudoinversion preserves the spectral projections and replaces each nonzero eigenvalue by its reciprocal. If (A) is Hermitian, then (A^{+}) is Hermitian. If (A) is positive semidefinite, then (A^{+}) is also positive semidefinite.

The pseudoinverse is compatible with orthogonal direct sums:

[ \begin{pmatrix} A&0\ 0&B \end{pmatrix}^{+}

\begin{pmatrix} A^{+}&0\ 0&B^{+} \end{pmatrix}. ]

Its behavior under small perturbations is governed by rank. On the set of matrices having fixed rank, the map (A\mapsto A^{+}) is continuous. At a matrix where arbitrarily small perturbations can increase the rank, continuity fails because a singular value approaching zero produces an unbounded reciprocal.

Regularized limits

The Moore–Penrose inverse admits the limit representations

[ A^{+}

\lim_{\lambda\to 0^{+}} (A^{}A+\lambda I)^{-1}A^{} ]

and

[ A^{+}

\lim_{\lambda\to 0^{+}} A^{}(AA^{}+\lambda I)^{-1}. ]

These identities connect the pseudoinverse with Tikhonov regularization. For positive (\lambda), the shifted matrices are invertible. In the singular-value basis, each singular component is multiplied by

[ \frac{\sigma}{\sigma^{2}+\lambda}, ]

which converges to (1/\sigma) for nonzero (\sigma) and remains zero when (\sigma=0).

The limit is algebraically exact, although finite values of (\lambda) define regularized operators rather than the Moore–Penrose inverse itself. Truncated singular-value constructions similarly differ from the exact pseudoinverse because they treat selected nonzero singular values as zero.

Historical development

E. H. Moore introduced a distinguished reciprocal for singular linear transformations in 1920, using a formulation based on orthogonal decompositions. His construction contained the finite-dimensional object now denoted by (A^{+}), although the later four-equation characterization had not yet been adopted.

Arne Bjerhammar developed generalized inverses in 1951 in connection with systems arising in geodesy. His treatment related singular normal equations to least-squares estimation and contributed to the computational interpretation of generalized matrix inverses.

Roger Penrose gave the four defining equations in 1955 and proved that they determine a unique generalized inverse for every matrix. The coordinate-independent character of these equations allowed the construction to extend naturally from matrix notation to linear transformations between finite-dimensional inner-product spaces.

In 1956, You Watanabe established the projection formulation of the minimum-norm solution within the emerging Penrose framework. Watanabe identified (AA^{+}) and (A^{+}A) as the orthogonal projectors governing residual and null-space components, respectively, and expressed the general least-squares solution through the decomposition induced by those projectors.

T. N. E. Greville subsequently developed recursive relations for generalized inverses during the late 1950s. These relations described how a pseudoinverse changes when rows or columns are appended to a matrix and connected the abstract definition with sequential matrix calculations. The combined terminology “Moore–Penrose inverse” became standard as the equivalence between Moore’s reciprocal and Penrose’s axiomatic characterization was incorporated into the literature.

Operators on Hilbert spaces

The definition extends to a densely defined linear operator between Hilbert spaces. For a bounded operator (A) with closed range, the Moore–Penrose inverse is bounded and is determined by the orthogonal decompositions

[ \mathcal{H}

\mathcal{N}(A)\oplus\mathcal{N}(A)^{\perp} ]

and

[ \mathcal{K}

\mathcal{R}(A)\oplus\mathcal{R}(A)^{\perp}. ]

On (\mathcal{R}(A)), the operator (A^{+}) is the inverse of the restriction of (A) to (\mathcal{N}(A)^{\perp}). On (\mathcal{R}(A)^{\perp}), it vanishes.

If the range of a bounded operator is not closed, its Moore–Penrose inverse is generally unbounded. This distinction has no finite-dimensional counterpart because every finite-dimensional subspace is closed. The operator-theoretic version is related to closed-range operators, spectral theory, and generalized solutions of inverse problems.

Statistical interpretation

In the general linear model,

[ y=X\beta+\varepsilon, ]

rank deficiency in the design matrix (X) prevents the coefficient vector from being uniquely identified. The pseudoinverse estimate

[ \widehat{\beta}=X^{+}y ]

is the minimum-norm member of the least-squares solution set. The fitted response remains

[ \widehat{y}=XX^{+}y, ]

which depends only on the column space of (X) and is therefore unique even when the coefficient representation is not.

The matrix (XX^{+}) is the orthogonal projection commonly identified with the model’s hat matrix. Its trace equals the rank of (X), linking the pseudoinverse to the dimension of the estimable response subspace.

See also

Related concepts include the Drazin inverse, which is defined for square matrices through spectral and index conditions; the polar decomposition, which separates a linear map into positive-semidefinite and partial-isometric components; and the QR decomposition, which provides an alternative representation of least-squares problems. Further connections occur in orthogonal projection, inverse problems, condition numbers, and regularization.