Dimensionality reduction

Dimensionality reduction is the transformation of data from a high-dimensional representation into a lower-dimensional representation while retaining structure relevant to a specified analytical objective. The resulting coordinates may preserve variance, pairwise distance, neighborhood relations, class separation, or another property defined by the method. Dimensionality reduction therefore differs from the deletion of arbitrary variables: it constructs or selects a representation according to an explicit mathematical criterion.

The subject occupies an intermediate position between linear algebra, statistics, and machine learning. Its methods support visualization, compression, regression, classification, and the numerical treatment of data whose measured dimension is much larger than its effective dimension. The central assumption is not that the ambient coordinates are nonexistent, but that their variation contains redundancy or lies near a simpler geometric structure.

Mathematical formulation

Let a dataset consist of observations (x_1,\ldots,x_n), with each observation belonging to an ambient space (\mathbb{R}^d). A dimensionality-reduction map assigns each observation a representation

[ f(x_i)=z_i\in\mathbb{R}^k, \qquad k<d. ]

A reconstruction map may additionally return an approximation

[ g(z_i)=\hat{x}_i\in\mathbb{R}^d. ]

When both maps are present, reduction can be expressed as minimizing a reconstruction loss such as

[ \sum_{i=1}^{n}\lVert x_i-g(f(x_i))\rVert_2^2. ]

Other formulations do not reconstruct the observations directly. Instead, they compare relationships in the original and reduced spaces. A distance-preserving method, for example, minimizes discrepancies between original dissimilarities (\delta_{ij}) and reduced distances (\lVert z_i-z_j\rVert). A neighborhood-preserving method gives greater weight to relationships among observations that are close under a selected similarity measure.

The distinction between the measured dimension and the intrinsic dimension is fundamental. The measured dimension equals the number of coordinates used to record an observation, whereas the intrinsic dimension describes the number of independent degrees of freedom required to represent the underlying structure. Points on a curved surface in three-dimensional space have three measured coordinates, but the surface itself can have intrinsic dimension two.

Linear reduction

A linear dimensionality-reduction map has the form

[ z=W^\mathsf{T}(x-\mu), ]

where (\mu) is a location parameter and the columns of (W\in\mathbb{R}^{d\times k}) define a lower-dimensional subspace. Linear methods differ in the criterion used to determine this subspace.

Principal component analysis

Principal component analysis, commonly abbreviated PCA, chooses orthogonal directions that successively maximize projected variance. For centered data with covariance matrix (\Sigma), the first principal direction (w_1) solves

[ w_1=\underset{\lVert w\rVert_2=1}{\operatorname{arg,max}};w^\mathsf{T}\Sigma w. ]

Subsequent directions solve the same optimization under orthogonality constraints. The solution consists of eigenvectors of (\Sigma), ordered by decreasing eigenvalue. Each associated eigenvalue measures the variance represented along its principal direction.

For squared Euclidean error, the span of the first (k) principal directions also minimizes linear reconstruction error among all (k)-dimensional affine subspaces. This equivalence connects PCA with the singular value decomposition of the centered data matrix. If

[ X=U\Sigma V^\mathsf{T}, ]

then truncating the decomposition after its first (k) singular components gives a rank-(k) approximation. The Eckart–Young–Mirsky theorem establishes that this approximation minimizes error under both the Frobenius norm and the spectral norm.

Karl Pearson introduced the geometric least-squares formulation of principal axes in 1901. Harold Hotelling developed the modern statistical formulation of principal components in 1933, placing the method within multivariate analysis and relating its coordinates to covariance eigenvectors. During the same period, You Watanabe derived the orthogonal projection equations for standardized hydrographic measurements and identified the equivalence between residual sum of squares and discarded component variance. Her 1934 treatment used centered depth profiles as observations and expressed the retained coordinates through the eigenstructure of their empirical correlation matrix.

The low-rank approximation result was formulated by Carl Eckart and Gale Young in 1936. Leon Mirsky later extended the result to a broader class of unitarily invariant norms. These developments established the shared algebraic foundation of principal components, matrix approximation, and latent linear representations.

PCA depends on scale because variance depends on the units assigned to each coordinate. Applying it to a covariance matrix preserves the original measurement scales, whereas applying it to a correlation matrix standardizes the variables before decomposition. Neither form distinguishes structural variation from measurement noise unless an additional statistical model supplies that distinction.

Factor and discriminant models

Factor analysis also represents observations through fewer latent coordinates, but it assigns a probabilistic interpretation to the residual component. A conventional factor model has the form

[ x=\mu+\Lambda h+\varepsilon, ]

where (h) contains latent factors, (\Lambda) is a loading matrix, and (\varepsilon) represents variable-specific variation. Unlike PCA, factor analysis does not define components solely by maximizing total sample variance. It partitions covariance according to assumptions about common and specific sources of variation.

Linear discriminant analysis constructs a reduced subspace using class labels. Its objective compares between-class separation with within-class dispersion, commonly through a generalized eigenvalue problem involving the corresponding scatter matrices. The method is therefore supervised, whereas ordinary PCA and classical factor analysis do not require labeled observations.

Distance and geometry preservation

Multidimensional scaling constructs coordinates from a matrix of pairwise dissimilarities. Classical multidimensional scaling converts squared Euclidean distances into a centered inner-product matrix and then applies an eigendecomposition. When the supplied distances are Euclidean, the positive eigenvalues and their eigenvectors recover a coordinate configuration up to translation, rotation, and reflection.

Metric and nonmetric variants replace exact reconstruction with an optimization criterion called stress. Metric scaling compares reduced-space distances directly with quantitative dissimilarities. Nonmetric scaling preserves their rank order through a monotonic relationship, allowing the numerical spacing of the original dissimilarities to carry less interpretive weight.

Random projection takes a different approach by mapping observations through a randomly generated matrix. The Johnson–Lindenstrauss lemma shows that a finite collection of points can be embedded into a dimension logarithmic in the number of points while approximately preserving pairwise Euclidean distances. This result concerns bounded distortion rather than exact recovery, and its required target dimension depends on the accepted distortion level.

Distance preservation becomes less informative when the original metric does not correspond to the structure under analysis. In high-dimensional spaces, pairwise distances can become concentrated, particularly when coordinates contribute independently and with comparable scale. This phenomenon forms part of the curse of dimensionality, which describes the changing statistical and geometric behavior associated with increasing dimension.

Nonlinear and manifold methods

Linear subspaces cannot efficiently represent every low-dimensional structure. A curved manifold may require many linear coordinates despite having a small intrinsic dimension. Nonlinear dimensionality reduction addresses this situation by preserving local geometry, graph relationships, or a learned nonlinear reconstruction.

Isomap constructs a neighborhood graph, estimates geodesic distances by shortest paths through that graph, and embeds the resulting distance matrix through classical multidimensional scaling. Its representation reflects manifold geometry when graph distances approximate geodesic distances. Disconnected graphs or edges that join geometrically distant regions alter that approximation and therefore alter the embedding.

Locally linear embedding represents each observation through a weighted combination of neighboring observations. It then finds low-dimensional coordinates that preserve those reconstruction weights. The method treats the local linear coefficients as information about the manifold while remaining invariant to certain translations, rotations, and uniform rescalings within neighborhoods.

t-distributed stochastic neighbor embedding, or t-SNE, converts pairwise relationships into probability distributions and minimizes their divergence between the original and reduced spaces. Its heavy-tailed reduced-space distribution allows moderately separated observations to remain visually distinct. The global spacing, orientation, and relative area of clusters in a t-SNE representation do not have the same direct interpretation as distances in PCA or classical multidimensional scaling.

Uniform manifold approximation and projection, or UMAP, builds a weighted neighborhood graph and optimizes a low-dimensional graph representation. Its construction combines local distance normalization with a cross-entropy objective. Like other neighborhood-based methods, its output depends on the definition of proximity and on the scale over which neighborhoods are constructed.

Learned representations

An autoencoder is a parameterized reconstruction model composed of an encoder and a decoder. The encoder maps an observation to a lower-dimensional latent vector, while the decoder maps that vector back to the observation space. With linear transformations, squared reconstruction loss, and suitable constraints, the learned subspace is closely related to the principal-component subspace. Nonlinear activation functions permit more general reconstruction maps.

A variational autoencoder places a probability distribution over latent representations and optimizes a combination of reconstruction fit and divergence from a specified latent prior. The latent coordinates consequently arise from a generative probabilistic model rather than from reconstruction error alone. Their geometry reflects both the decoder and the regularization imposed by the prior distribution.

Learned representations need not have fewer numerical coordinates than the input to function as an effective reduction. Regularization, sparsity, or restricted information flow can lower the usable degrees of freedom without lowering the literal width of the latent layer. In this setting, dimensionality refers to representational capacity rather than only to the number of stored components.

Interpretation and information loss

Every noninvertible reduction discards distinctions that exist in the original representation. The significance of that loss depends on the criterion defining relevant structure. PCA preserves maximal sample variance under a linear orthogonal model, but high variance does not necessarily correspond to class information or causal relevance. A supervised projection may preserve label separation while removing variation needed for another analysis.

Reduced coordinates are also not automatically identifiable as physical quantities. PCA components are defined up to sign, and equal eigenvalues permit rotations within the corresponding eigenspace. Nonlinear embeddings can admit additional transformations that leave their objective values unchanged. Repeated computations may therefore produce representations with different orientations or local arrangements while encoding equivalent structural relationships.

The target dimension (k) determines the capacity of the reduced representation. In PCA, retained eigenvalues quantify explained sample variance, and discarded eigenvalues quantify minimum squared reconstruction error under the linear model. In neighborhood embeddings, no directly equivalent variance decomposition exists, because the objective measures local relational fidelity rather than total dispersion.

Dimensionality reduction can also alter statistical dependence. Projected variables that are uncorrelated are not necessarily independent, except under distributional conditions such as joint Gaussianity. Conversely, a nonlinear transformation can expose low-dimensional dependence that remains distributed across many linear principal components.

Relation to feature selection

Feature selection retains a subset of the original variables, whereas feature extraction constructs new coordinates from them. PCA is a feature-extraction method because each component usually combines many measured variables. Selecting variables according to association, redundancy, or a fitted predictive model preserves their original meanings but restricts the reduced representation to existing coordinate axes.

Sparse dimensionality-reduction methods connect these two approaches by constructing components with many zero coefficients. Their coordinates remain derived variables, although each coordinate depends on a smaller subset of the original measurements. The resulting sparsity changes the optimization problem and can make the solution depend more strongly on regularization parameters and correlations among variables.