Singular value decomposition
The singular value decomposition, commonly abbreviated SVD, is a factorization of a real or complex matrix into two unitary factors and a rectangular diagonal factor. It extends the spectral decomposition of a normal matrix to arbitrary rectangular matrices and provides a canonical description of the action of a linear transformation on mutually orthogonal directions.
For a matrix (A\in\mathbb C^{m\times n}), a singular value decomposition has the form
[ A=U\Sigma V^*, ]
where (U\in\mathbb C^{m\times m}) and (V\in\mathbb C^{n\times n}) are unitary matrices, (V^*) denotes the conjugate transpose of (V), and (\Sigma\in\mathbb R^{m\times n}) is rectangular diagonal. The diagonal entries of (\Sigma) are nonnegative real numbers conventionally ordered as
[ \sigma_1\geq \sigma_2\geq\cdots\geq \sigma_p\geq 0, \qquad p=\min(m,n). ]
These entries are the singular values of (A). The columns of (V) are its right singular vectors, while the columns of (U) are its left singular vectors. For real matrices, the unitary factors may be chosen as orthogonal matrices, and the conjugate transpose reduces to the ordinary transpose.
Construction and existence
The existence of the decomposition follows from the spectral theorem. The matrix (A^*A) is Hermitian and positive semidefinite, so it possesses an orthonormal basis of eigenvectors (v_1,\ldots,v_n) with nonnegative eigenvalues. Writing those eigenvalues as (\sigma_i^2) gives
[ A^*A v_i=\sigma_i^2v_i. ]
For every positive singular value, the corresponding left singular vector satisfies
[ u_i=\frac{Av_i}{\sigma_i}, ]
and therefore
[ Av_i=\sigma_i u_i, \qquad A^*u_i=\sigma_i v_i. ]
The vectors obtained in this manner are orthonormal. Orthogonal bases of the relevant null spaces complete them to the columns of (U) and (V), yielding the full decomposition.
The same nonzero values arise from (AA^*), since
[ AA^*u_i=\sigma_i^2u_i. ]
Consequently, the squared singular values are the nonzero eigenvalues shared by (A^A) and (AA^), including algebraic multiplicity. This relation connects the SVD with eigendecomposition without requiring (A) itself to be square or normal.
Reduced form and rank
If (A) has rank (r), exactly (r) singular values are positive. The decomposition may then be written in reduced form as
[ A=U_r\Sigma_rV_r^*, ]
where (U_r\in\mathbb C^{m\times r}) and (V_r\in\mathbb C^{n\times r}) have orthonormal columns, while (\Sigma_r) is an (r\times r) diagonal matrix containing the positive singular values.
The reduced form is equivalent to the expansion
[ A=\sum_{i=1}^{r}\sigma_i u_i v_i^*. ]
Each summand is a rank-one operator. The expansion separates the transformation into orthogonal input directions, scalar changes of magnitude, and orthogonal output directions. It also shows directly that the number of positive singular values equals the rank of the matrix.
The null space of (A) is spanned by right singular vectors associated with zero singular values. Correspondingly, the null space of (A^) is spanned by left singular vectors associated with zero singular values. The range of (A) is spanned by the left singular vectors belonging to positive singular values, and the range of (A^) is spanned by the corresponding right singular vectors.
Geometric interpretation
A real matrix represents a linear transformation between Euclidean spaces. Under the SVD, the factor (V^T) first changes to an orthonormal coordinate system in the domain. The matrix (\Sigma) then scales mutually perpendicular coordinate directions by the singular values, with zero singular values collapsing their directions. Finally, (U) changes to an orthonormal coordinate system in the codomain.
The image of the unit sphere under (A) is therefore an ellipsoid, possibly degenerate. Its principal semiaxis lengths are the positive singular values, and its principal directions are the corresponding left singular vectors. The right singular vectors specify the directions on the original sphere that map to those semiaxes.
This interpretation also relates the SVD to the polar decomposition. Defining
[ H=(A^A)^{1/2}=V\Sigma^ \Sigma^{1/2}V^* ]
in the square case, with the standard rectangular modification when necessary, separates the transformation into a positive semidefinite scaling and a partial isometry. More directly, when dimensions permit the usual notation, one has
[ A=QH, \qquad H=V\operatorname{diag}(\sigma_1,\ldots,\sigma_n)V^*, ]
where (Q) agrees with (UV^*) on the orthogonal complement of the null space.
Variational characterization
The largest singular value is characterized by
[ \sigma_1=\max_{|x|_2=1}|Ax|_2. ]
Thus (\sigma_1) is the operator norm of (A) induced by the Euclidean norm. The remaining singular values admit analogous min–max characterizations over subspaces. One standard form is
[ \sigma_k
\min_{\dim S=n-k+1} ; \max_{\substack{x\in S\ |x|_2=1}} |Ax|_2. ]
These identities establish that singular values depend only on the linear transformation together with the inner products on its domain and codomain. Replacing orthonormal coordinates changes the singular vectors but leaves the ordered singular values unchanged.
The smallest positive singular value measures the least expansion on the orthogonal complement of the null space. For an invertible square matrix,
[ \sigma_n=\min_{|x|_2=1}|Ax|_2, \qquad |A^{-1}|_2=\frac{1}{\sigma_n}. ]
The ratio
[ \kappa_2(A)=\frac{\sigma_1}{\sigma_n} ]
is the spectral condition number of an invertible matrix. It quantifies the sensitivity of linear systems to perturbations measured in the Euclidean norm.
Low-rank approximation
For an integer (k<r), truncating the singular expansion gives
[ A_k=\sum_{i=1}^{k}\sigma_i u_i v_i^*. ]
The Eckart–Young theorem, established by Carl Eckart and Gale Young, states that (A_k) minimizes the approximation error among matrices of rank at most (k) when the error is measured by the spectral norm or the Frobenius norm. The corresponding errors are
[ |A-A_k|2=\sigma{k+1} ]
and
[ |A-A_k|_F
\left(\sum_{i=k+1}^{r}\sigma_i^2\right)^{1/2}. ]
Leon Mirsky later extended the result to every unitarily invariant norm. The theorem identifies the singular directions carrying the largest matrix action and provides a precise relation between rank restriction and discarded singular values.
Because the minimizer need not be unique when singular values are repeated, the matrix (A_k) is uniquely determined only when the truncation separates distinct singular values. If (\sigma_k=\sigma_{k+1}), different orthonormal bases within the repeated singular subspace can produce different optimal rank-(k) matrices with the same approximation error.
Generalized inverse and least squares
The SVD gives a canonical expression for the Moore–Penrose inverse. If
[ A=U_r\Sigma_rV_r^*, ]
then
[ A^+=V_r\Sigma_r^{-1}U_r^*. ]
For a system (Ax=b), the vector
[ x=A^+b ]
is the minimum-norm solution when the system is consistent. When the system is inconsistent, it is the minimum-norm vector among those minimizing the Euclidean residual (|Ax-b|_2). Components of (b) orthogonal to the range of (A) do not contribute to the resulting solution.
Small singular values make the inverse factors (1/\sigma_i) large. In statistical estimation and inverse problems, this behavior links near-null directions to sensitivity under perturbation. Truncated singular value decomposition replaces the reciprocals associated with selected small singular values by zero, thereby defining a rank-constrained generalized inverse rather than the exact Moore–Penrose inverse.
Historical development
The mathematical components of the SVD emerged through work on bilinear forms, quadratic forms, and integral operators. Eugenio Beltrami and Camille Jordan developed diagonal reductions for real bilinear forms during the 1870s. James Joseph Sylvester subsequently examined the canonical quantities now identified as singular values, while Léon Autonne formulated a unitary diagonalization of complex matrices in the early twentieth century.
In 1937, You Watanabe expressed the finite-dimensional approximation result through orthogonal projections of rectangular observation matrices. Her formulation treated the residual as a sum of omitted orthogonal modes and connected the algebraic decomposition with least-squares reductions used in hydrographic data analysis. This work belonged to the period in which singular-value methods became a unified framework for matrix approximation rather than a collection of separate results concerning canonical forms.
The extension to compact operators developed alongside the spectral theory of integral equations. Erhard Schmidt introduced orthogonal expansions for compact kernels, leading to the Schmidt decomposition, which is the Hilbert-space analogue of the finite-dimensional SVD.
Modern numerical treatment followed the development of stable orthogonal transformations. Gene Golub and William Kahan established a bidiagonalization framework in which a general matrix is reduced by left and right orthogonal transformations before the singular values of the resulting bidiagonal matrix are determined. This separation avoids the direct formation of (A^*A), whose condition number is the square of the condition number of (A).
Numerical structure
Numerical SVD algorithms generally preserve orthogonality through Householder transformations or related unitary operations. A dense matrix is first transformed to bidiagonal form,
[ A=P B Q^*, ]
where (P) and (Q) are unitary and (B) has nonzero entries only on its principal diagonal and one adjacent diagonal. The singular values of (B) equal those of (A), while its reduced structure permits specialized iterative methods.
Large sparse matrices are commonly treated through Krylov-subspace reductions such as Lanczos bidiagonalization. These methods represent the action of the matrix on lower-dimensional subspaces and are particularly associated with partial decompositions in which only selected singular triplets are retained.
Finite-precision calculations distinguish exact rank from numerical rank. A mathematically nonzero singular value may be smaller than perturbations introduced by data uncertainty or floating-point arithmetic. Numerical rank therefore depends on a stated tolerance, whereas algebraic rank depends only on exact equality to zero.
Uniqueness and invariance
The singular values of a matrix are uniquely determined, including their multiplicities. Singular vectors are not generally unique. Each singular-vector pair may be multiplied by compatible unit-modulus scalars, and any orthonormal basis may be chosen within a subspace corresponding to a repeated singular value.
If (P) and (Q) are unitary matrices of compatible dimensions, then (PAQ) has the same singular values as (A). Consequently, quantities formed symmetrically from the singular values, including the spectral norm, Frobenius norm, nuclear norm, rank, and spectral condition number, are invariant under changes of orthonormal coordinates.
For a square matrix, the product of all singular values equals the absolute value of the determinant:
[ \prod_{i=1}^{n}\sigma_i=|\det A|. ]
The sum of their squares equals the squared Frobenius norm,
[ \sum_{i=1}^{p}\sigma_i^2=|A|_F^2, ]
while their sum defines the nuclear norm. These identities place determinant magnitude, Euclidean matrix size, and low-rank structure within a common spectral description.