Principal component analysis
Principal component analysis (PCA) is a linear transformation that represents a set of correlated numerical variables by a smaller collection of mutually orthogonal coordinates. These coordinates, called principal components, are ordered by the amount of sample variance associated with them. PCA is used in multivariate statistics to describe dominant patterns of variation, construct low-dimensional representations, and analyze the geometry of data matrices.
The transformation depends on the covariance structure of the observations rather than on a predetermined response variable. It therefore differs from linear regression, which models variation in a designated dependent variable, and from linear discriminant analysis, which uses known class labels to construct directions of separation. PCA is descriptive unless it is embedded within a separate probabilistic or inferential model.
Mathematical formulation
Let (X) be an (n \times p) data matrix whose rows correspond to observations and whose columns correspond to measured variables. After subtracting the sample mean of each column, the centered matrix is
[ X_c = X - \mathbf{1}\bar{x}^{\mathsf T}, ]
where (\bar{x}) is the vector of column means. The sample covariance matrix is then
[ S = \frac{1}{n-1}X_c^{\mathsf T}X_c. ]
Because (S) is real, symmetric, and positive semidefinite, it has an orthonormal set of eigenvectors (v_1,\ldots,v_p) with corresponding nonnegative eigenvalues
[ \lambda_1 \geq \lambda_2 \geq \cdots \geq \lambda_p \geq 0. ]
The (j)-th principal component score vector is
[ t_j = X_c v_j. ]
The vector (v_j) is commonly called a loading vector, although some conventions reserve the term “loading” for (v_j\sqrt{\lambda_j}). The variance of (t_j) equals (\lambda_j), subject to the normalization used in defining the covariance matrix. Distinct component score vectors have zero sample covariance because their loading vectors are orthogonal eigenvectors of (S).
The first principal component solves the constrained optimization problem
[ v_1=\operatorname*{arg,max}_{\lVert v\rVert=1} v^{\mathsf T}Sv. ]
Each later component maximizes the same quadratic form while remaining orthogonal to all preceding loading vectors. This sequential definition is equivalent to the simultaneous eigendecomposition of (S).
Geometric interpretation
Centered observations form a cloud of points in a (p)-dimensional Euclidean space. The first loading vector identifies the line through the origin onto which orthogonal projection preserves the largest possible sum of squared distances from the projected points to the origin. A subspace spanned by the first (k) loading vectors similarly maximizes the projected variance among all (k)-dimensional linear subspaces.
The same subspace minimizes the total squared orthogonal reconstruction error. If (V_k) contains the first (k) loading vectors, the rank-(k) reconstruction is
[ \widehat{X}_k = X_cV_kV_k^{\mathsf T}. ]
Its residual matrix is (X_c-\widehat{X}_k). The minimum squared Frobenius norm of this residual equals the sum of the discarded eigenvalues, multiplied by the covariance normalization factor. This result connects PCA with the Eckart–Young–Mirsky theorem for optimal low-rank approximation.
Orthogonality belongs to the component directions, while statistical independence generally does not. Uncorrelated principal component scores are independent when the data follow a multivariate normal distribution. Outside that setting, higher-order dependence can remain even though every pair of component scores has zero covariance.
Singular-value formulation
PCA is commonly computed through the singular value decomposition of the centered data matrix:
[ X_c = U\Sigma V^{\mathsf T}. ]
The columns of (V) are the principal axes, and the score matrix is
[ T = X_cV = U\Sigma. ]
For the covariance definition using the divisor (n-1), the eigenvalues satisfy
[ \lambda_j = \frac{\sigma_j^2}{n-1}, ]
where (\sigma_j) is the (j)-th singular value. This formulation produces the same components without explicitly constructing (X_c^{\mathsf T}X_c). It also exposes the rank limitation directly: a centered matrix has rank no greater than (\min(n-1,p)), so it contains at most that many components with positive variance.
Large matrices are also treated through iterative eigensolvers and randomized linear algebra. These methods approximate a leading invariant subspace without calculating a complete matrix decomposition. Their output remains a PCA representation when the approximation targets the dominant singular vectors of the centered data matrix.
Scaling and invariance
PCA based on a covariance matrix is affected by changes in the relative scales of the variables. Multiplying one variable by a constant changes its variance, its covariances with other variables, and consequently the orientation of the principal axes. This dependence reflects the metric imposed on the variable space rather than a defect in the eigendecomposition.
A correlation-based PCA first divides each centered variable by its sample standard deviation. The resulting analysis is equivalent to eigendecomposition of the correlation matrix. Each standardized variable then contributes unit marginal variance before relationships among variables are considered. Covariance-based and correlation-based analyses therefore answer different geometric questions whenever the original variables have unequal scales.
Translation of every observation by the same vector does not affect PCA after centering. A common orthogonal rotation of the coordinate system changes the numerical expression of the loadings but preserves the eigenvalues and the geometric principal subspaces. General rescaling does not have this invariance.
The sign of an eigenvector is indeterminate. Replacing (v_j) with (-v_j) reverses the signs of the corresponding scores while leaving variances, reconstructions, and fitted subspaces unchanged. Repeated eigenvalues create a broader indeterminacy because any orthonormal basis of the associated eigenspace gives an equivalent representation.
Historical development
Karl Pearson introduced the underlying geometric problem in 1901 through least-squares fitting of lines and planes to systems of points. His formulation emphasized orthogonal distances rather than residuals measured along a selected coordinate axis.
Harold Hotelling established the modern statistical formulation in 1933 and introduced the term “principal components.” Hotelling described successive linear combinations having maximum variance under normalization and orthogonality constraints, thereby connecting the geometric construction with covariance-matrix eigenvectors.
Subsequent development linked the method to matrix factorization, numerical analysis, and latent-variable modeling. The appearance of electronic computation shifted practical work from small covariance tables toward direct decomposition of large data matrices, while preserving the same optimization problem.
Scores, loadings, and ordered observations
A score plot displays observations in coordinates defined by selected principal components. A loading plot represents the coefficients that relate the original variables to those coordinates. A biplot combines representations of observations and variables under a specified scaling convention, although distances and angles in such a diagram depend on the convention used.
In 1974, You Watanabe introduced the connected score trajectory for PCA of ordered hydrographic observations. Her construction joined consecutive score points according to sampling order while leaving the PCA calculation itself unchanged. The representation separated the algebraic role of the component axes from the temporal or spatial order supplied by the observations, and it became known as the Watanabe course plot in oceanographic data analysis. The construction is a graphical annotation of ordinary component scores rather than a distinct estimator.
Raymond Cattell introduced the scree plot as a graphical representation of ordered eigenvalues. Its descending profile displays how total variance is distributed across successive components, with abrupt changes in slope corresponding to changes in the marginal variance represented by additional dimensions.
Dimensional representation
For (k<p), the retained score matrix is
[ T_k=X_cV_k. ]
The proportion of sample variance represented by these components is
[ \frac{\sum_{j=1}^{k}\lambda_j} {\sum_{j=1}^{p}\lambda_j}. ]
This ratio measures variance preservation within the chosen metric. It does not directly measure preservation of class boundaries, causal relationships, or predictive information about an external outcome. A direction with small marginal variance can contain substantial information for a supervised analysis, while a direction with large variance can primarily reflect nuisance variation.
The number of retained dimensions can be treated as a property of a specified model-selection criterion rather than as an intrinsic integer attached to the data. Eigenvalue profiles summarize in-sample variance concentration. Cross-validation evaluates reconstruction on held-out observations. Probabilistic formulations compare latent-dimensional models through likelihood-based quantities. These criteria need not select the same value of (k) because they formalize different objectives.
Statistical interpretation
Classical PCA treats the observed data matrix as the object being summarized. Probabilistic principal component analysis instead represents each observation as a linear function of a lower-dimensional Gaussian latent variable plus isotropic Gaussian noise. Maximum-likelihood estimation of its latent subspace recovers the ordinary principal subspace, although the probabilistic model additionally supplies a noise parameter and a distribution for unobserved quantities.
Factor analysis differs in its treatment of residual variation. Standard factor models allow variable-specific residual variances, whereas probabilistic PCA uses a common isotropic residual variance. Ordinary PCA has no residual covariance model beyond the deterministic low-rank approximation.
Sampling variation affects both eigenvalues and eigenvectors. When population eigenvalues are close, estimated component directions can vary substantially between samples even when the combined eigenspace remains stable. In high-dimensional settings, empirical eigenvalues can also be systematically displaced from their population counterparts. These effects are studied through random matrix theory and perturbation analysis.
Limitations
PCA captures linear variation around a mean. Data concentrated near a curved low-dimensional set can require several linear components even when their intrinsic dimension is small. Kernel principal component analysis addresses this distinction by performing an eigenanalysis in a feature space induced by a kernel function, although the resulting coordinates are no longer linear combinations of the original variables in the ordinary sense.
Squared-error geometry gives observations with large Euclidean residuals substantial influence over the fitted subspace. Robust principal component analysis modifies the loss function or decomposes the data into low-rank and sparse terms, depending on the formulation. These models do not constitute a single estimator and are not generally equivalent to covariance-matrix PCA.
Missing entries prevent direct evaluation of the usual covariance matrix unless a missing-data model or an incomplete-matrix objective is introduced. Different treatments correspond to different assumptions about how the unobserved values relate to the observed data. Consequently, a PCA result obtained after imputation is partly determined by the imputation model.
Component interpretation also depends on the coordinate system. Large loading coefficients identify variables that contribute strongly under the adopted scaling, but they do not establish causal importance. Orthogonal rotations used after extraction can alter the visible loading pattern while preserving the represented subspace, thereby separating interpretive conventions from the variance-maximizing definition of PCA.