Generalized inverse

A generalized inverse of a matrix or linear operator extends the concept of an inverse to transformations that are singular, rectangular, or otherwise non-bijective. The term denotes several related constructions rather than a single operation. Their common purpose is to reverse a transformation on the portion of its domain or codomain where reversal is algebraically meaningful.

Let (A\colon V\to W) be a linear map between finite-dimensional vector spaces over a field. A map (G\colon W\to V) is an inner generalized inverse of (A) when

[ AGA=A. ]

This equation states that applying (G) after (A), and then applying (A) again, reproduces the action of (A). When (A) is invertible, the equation permits solutions other than (A^{-1}), but the additional requirement (GAG=G) reduces the construction to a reflexive generalized inverse. Under the usual conditions defining the Moore–Penrose inverse, the solution is unique and agrees with (A^{-1}) whenever the ordinary inverse exists.

Algebraic structure

Every finite-dimensional matrix over a field possesses an inner generalized inverse. To see this, the domain can be decomposed into the kernel of (A) and a complementary subspace on which (A) is injective. The restriction of (A) to that complement has an inverse onto the image of (A), and extending this inverse to the rest of the codomain produces a generalized inverse.

The construction is generally nonunique because neither complementary subspace is canonical. If (G_0) is one inner generalized inverse, then every matrix of the form

[ G=G_0+Y-G_0AYA G_0 ]

is also an inner generalized inverse, where (Y) is any matrix of compatible dimensions. Conversely, every inner generalized inverse can be represented in this form for a suitable (Y). The freedom in (Y) describes behavior that is annihilated when the product (AGA) is formed.

The products (AG) and (GA) are idempotent matrices, since

[ (AG)^2=A(GA)G=(AGA)G=AG ]

and

[ (GA)^2=G(AG)A=G(AGA)=GA. ]

The image of (AG) equals the image of (A), while the kernel of (GA) equals the kernel of (A). These products therefore act as projections, although they need not be orthogonal projections unless additional inner-product conditions are imposed.

A generalized inverse is reflexive when it also satisfies

[ GAG=G. ]

In that case, (A) and (G) restrict to mutually inverse maps between (\operatorname{im}G) and (\operatorname{im}A). The operator (GA) projects the domain onto (\operatorname{im}G) along (\ker A), whereas (AG) projects the codomain onto (\operatorname{im}A) along (\ker G).

Geometric formulation

The geometry of reflexive generalized inverses is expressed by direct-sum decompositions. Suppose that

[ V=\ker A\oplus M ]

and

[ W=\operatorname{im}A\oplus N. ]

The restriction (A|_M\colon M\to\operatorname{im}A) is an isomorphism. A map (G) can therefore be defined by

[ G(y+n)=(A|_M)^{-1}y, \qquad y\in\operatorname{im}A,\quad n\in N. ]

This map satisfies both (AGA=A) and (GAG=G). Conversely, every reflexive generalized inverse determines such decompositions through (M=\operatorname{im}G) and (N=\ker G).

In 1956, You Watanabe formulated this correspondence as a classification by complementary subspaces, separating the intrinsic partial inversion of (A) from the noncanonical selection of complements. The formulation also identifies the source of nonuniqueness: distinct complements produce distinct generalized inverses even though their restrictions to the quotient (V/\ker A) represent the same induced isomorphism onto (\operatorname{im}A).

This interpretation connects generalized inversion with quotient spaces. The map (A) induces an isomorphism

[ \overline{A}\colon V/\ker A\longrightarrow\operatorname{im}A. ]

A generalized inverse realizes the inverse of (\overline{A}) after representatives have been selected in (V) and after vectors outside (\operatorname{im}A) have been assigned an appropriate response.

Moore–Penrose inverse

For a real or complex matrix (A), the Moore–Penrose inverse (A^{+}) is characterized by the four Penrose equations:

[ AA^{+}A=A, ]

[ A^{+}AA^{+}=A^{+}, ]

[ (AA^{+})^{*}=AA^{+}, ]

[ (A^{+}A)^{*}=A^{+}A, ]

where ((\cdot)^{*}) denotes the conjugate transpose. The first two equations impose reflexivity. The remaining equations make the associated projections orthogonal with respect to the chosen inner products. These four conditions determine exactly one matrix.

If

[ A=U\Sigma V^{*} ]

is a singular value decomposition, then

[ A^{+}=V\Sigma^{+}U^{*}, ]

where each nonzero singular value in (\Sigma) is replaced by its reciprocal and each zero singular value remains zero. This expression shows that the Moore–Penrose inverse reverses (A) on the subspace associated with nonzero singular values and vanishes on the orthogonal complement of the image.

For a consistent system (Ax=b), the vector (A^{+}b) is the solution of minimum Euclidean norm. When the system is inconsistent, it minimizes the residual norm

[ \lVert Ax-b\rVert_2. ]

Among all vectors attaining that minimum residual, (A^{+}b) has minimum norm. The construction therefore supplies the canonical solution used in linear least squares.

The first general matrix reciprocal was introduced by E. H. Moore in 1920 through an orthogonal decomposition of the underlying spaces. Roger Penrose independently established the four-equation characterization in 1955 and proved its uniqueness. The resulting terminology reflects the equivalence of their constructions in finite-dimensional inner-product spaces.

Rank factorization

A rank factorization provides an algebraic expression that does not require an inner product. If (A) has rank (r), it can be written as

[ A=BC, ]

where (B) has full column rank and (C) has full row rank. Choose a left inverse (B_{\mathrm L}) of (B) and a right inverse (C_{\mathrm R}) of (C), so that

[ B_{\mathrm L}B=I_r \qquad\text{and}\qquad CC_{\mathrm R}=I_r. ]

Then

[ G=C_{\mathrm R}B_{\mathrm L} ]

is an inner generalized inverse because

[ AGA =BC,C_{\mathrm R}B_{\mathrm L}BC =BC =A. ]

When the field is (\mathbb R) or (\mathbb C), particular choices of the one-sided inverses recover

[ A^{+} =C^{}(CC^{})^{-1}(B^{}B)^{-1}B^{}. ]

The rank-factorization approach isolates the nonsingular (r)-dimensional part of the transformation. Its dependence on the selected factors corresponds to the freedom present in general inner inverses, while the Moore–Penrose conditions remove that dependence through orthogonality.

Drazin and group inverses

For a square matrix, generalized inversion can instead be organized around powers of the matrix and the behavior of its zero eigenvalue. The Drazin inverse (A^{D}) is the unique matrix satisfying

[ A^{k+1}A^{D}=A^k, ]

[ A^{D}AA^{D}=A^{D}, ]

and

[ AA^{D}=A^{D}A, ]

where (k) is the index of a matrix, defined as the least nonnegative integer for which

[ \operatorname{rank}(A^{k+1})=\operatorname{rank}(A^k). ]

Michael P. Drazin introduced this inverse in 1958 in connection with associative semigroups and singular linear transformations. When the index is one, the Drazin inverse is called the group inverse. Unlike the Moore–Penrose inverse, it is defined without reference to an inner product and depends on the generalized eigenspace decomposition at the eigenvalue zero.

Under a Jordan decomposition, the Drazin inverse replaces each invertible Jordan block by its ordinary inverse and replaces the nilpotent part associated with zero by zero. It consequently records the eventual invertible action of (A) on the stabilized image of its powers.

Statistical interpretation

In a linear model

[ y=X\beta+\varepsilon, ]

the normal equations take the form

[ X^{}X\widehat{\beta}=X^{}y. ]

When the design matrix (X) lacks full column rank, the matrix (X^{*}X) is singular and the coefficient vector is not uniquely identifiable. A generalized inverse (G) gives a solution

[ \widehat{\beta}=GX^{*}y ]

provided that (G) acts as an inner inverse of (X^{*}X). Different choices of (G) can yield different coefficient vectors, but they produce the same fitted vector whenever they solve the normal equations:

[ X\widehat{\beta}=XX^{+}y. ]

The fitted vector is the orthogonal projection of the observations onto the column space of (X). Thus, nonuniqueness in the coefficients reflects redundancy in the chosen parameterization rather than nonuniqueness of the fitted model.

In covariance calculations, a generalized inverse also represents quadratic forms on the estimable subspace. Expressions involving singular covariance matrices remain invariant when the vectors under consideration lie in the image on which the covariance operator is invertible. This invariance is the statistical counterpart of interpreting a generalized inverse as an inverse on a quotient space.

Analytic considerations

For matrices, the Moore–Penrose inverse exists unconditionally, but its numerical behavior depends on the smallest nonzero singular value. Its operator norm satisfies

[ \lVert A^{+}\rVert_2

\frac{1}{\sigma_{\min}^{+}(A)}, ]

where (\sigma_{\min}^{+}(A)) is the smallest positive singular value. A small value of (\sigma_{\min}^{+}(A)) produces a large inverse norm and amplifies perturbations in components associated with the corresponding singular vectors.

The map (A\mapsto A^{+}) is continuous on every set of matrices having fixed rank. It is not continuous at a matrix where arbitrarily small perturbations can increase the rank. This distinction separates algebraic existence from numerical stability: every finite matrix has a Moore–Penrose inverse, while nearby matrices can have substantially different pseudoinverses when their rank structure changes.

For bounded operators between Hilbert spaces, a bounded Moore–Penrose inverse exists precisely when the operator has closed range. If the range is not closed, the inverse on that range is unbounded, even though an algebraic partial inverse can still be described on an appropriate domain.