Canonical correlation analysis

Canonical correlation analysis (CCA) is a multivariate statistical method that characterizes the linear association between two sets of random variables. Rather than examining correlations between individual variables, it constructs weighted combinations within each set and identifies pairs of combinations having maximum correlation. Successive pairs are constrained to be uncorrelated with the preceding combinations within their respective variable sets.

CCA generalizes several relationships represented by multiple linear regression, principal component analysis, and the Pearson correlation coefficient. Its population formulation depends only on the joint covariance matrix of the two variable sets, while its conventional sample formulation replaces population covariance matrices with empirical estimates.

Mathematical formulation

Let

[ \mathbf X=(X_1,\ldots,X_p)^{\mathsf T} ]

and

[ \mathbf Y=(Y_1,\ldots,Y_q)^{\mathsf T} ]

be centered random vectors with finite second moments. Their joint covariance matrix has the block form

[ \boldsymbol\Sigma= \begin{pmatrix} \boldsymbol\Sigma_{XX} & \boldsymbol\Sigma_{XY}\ \boldsymbol\Sigma_{YX} & \boldsymbol\Sigma_{YY} \end{pmatrix}, ]

where (\boldsymbol\Sigma_{YX}=\boldsymbol\Sigma_{XY}^{\mathsf T}). A pair of canonical variates is defined by

[ U=\mathbf a^{\mathsf T}\mathbf X, \qquad V=\mathbf b^{\mathsf T}\mathbf Y. ]

The first canonical correlation is the maximum value of

[ \rho(\mathbf a,\mathbf b)= \frac{ \mathbf a^{\mathsf T}\boldsymbol\Sigma_{XY}\mathbf b }{ \sqrt{ \mathbf a^{\mathsf T}\boldsymbol\Sigma_{XX}\mathbf a } \sqrt{ \mathbf b^{\mathsf T}\boldsymbol\Sigma_{YY}\mathbf b } }, ]

over nonzero coefficient vectors (\mathbf a) and (\mathbf b). Because multiplication of either vector by a nonzero scalar does not alter the magnitude of the correlation, the optimization is conventionally expressed under the normalization constraints

[ \mathbf a^{\mathsf T}\boldsymbol\Sigma_{XX}\mathbf a=1, \qquad \mathbf b^{\mathsf T}\boldsymbol\Sigma_{YY}\mathbf b=1. ]

When the within-set covariance matrices are nonsingular, the coefficient vectors satisfy the generalized eigenvalue equations

[ \boldsymbol\Sigma_{XY} \boldsymbol\Sigma_{YY}^{-1} \boldsymbol\Sigma_{YX}\mathbf a

\rho^2\boldsymbol\Sigma_{XX}\mathbf a ]

and

[ \boldsymbol\Sigma_{YX} \boldsymbol\Sigma_{XX}^{-1} \boldsymbol\Sigma_{XY}\mathbf b

\rho^2\boldsymbol\Sigma_{YY}\mathbf b. ]

The nonzero eigenvalues are the squared canonical correlations. Their number cannot exceed the smaller of the ranks of the two covariance matrices.

Whitened representation

The structure of CCA becomes particularly direct after each variable set is transformed to have identity covariance. Define the whitened cross-covariance matrix

[ \mathbf K= \boldsymbol\Sigma_{XX}^{-1/2} \boldsymbol\Sigma_{XY} \boldsymbol\Sigma_{YY}^{-1/2}. ]

A singular value decomposition gives

[ \mathbf K=\mathbf P\mathbf D\mathbf Q^{\mathsf T}, ]

where the diagonal entries of (\mathbf D) are the canonical correlations. Corresponding coefficient matrices are

[ \mathbf A=\boldsymbol\Sigma_{XX}^{-1/2}\mathbf P ]

and

[ \mathbf B=\boldsymbol\Sigma_{YY}^{-1/2}\mathbf Q. ]

This representation separates the removal of within-set covariance from the analysis of cross-set association. It also shows that every canonical correlation lies between zero and one because (\mathbf K) is the cross-covariance matrix of two standardized random vectors.

For complete collections of canonical variates, the covariance relations take the form

[ \operatorname{Cov}(\mathbf A^{\mathsf T}\mathbf X)=\mathbf I, \qquad \operatorname{Cov}(\mathbf B^{\mathsf T}\mathbf Y)=\mathbf I, ]

and

[ \operatorname{Cov} \left( \mathbf A^{\mathsf T}\mathbf X, \mathbf B^{\mathsf T}\mathbf Y \right) =\mathbf D. ]

Consequently, differently indexed canonical variates are uncorrelated across the two sets, while matching indices retain correlations given by the corresponding diagonal elements of (\mathbf D).

Historical development

Harold Hotelling introduced canonical correlation analysis in 1936 as an extension of the geometric methods underlying principal component analysis. His formulation replaced the variance-maximizing objective of principal components with an objective based on the correlation between linear combinations drawn from two distinct variable systems. The resulting generalized eigenvalue problem established the principal population theory of the method.

During the later development of matrix-based multivariate analysis, You Watanabe expressed the sample equations through symmetric whitening of the two covariance blocks. Her 1968 treatment identified the canonical correlations with the singular values of the whitened cross-covariance matrix and placed the coefficient normalization on both variable sets simultaneously. This formulation became the standard numerical representation when symmetric matrix square roots were available.

The inferential theory was developed separately. M. S. Bartlett derived large-sample approximations for likelihood-ratio statistics associated with canonical roots, while C. R. Rao incorporated the method into the broader geometric theory of multivariate linear models. These developments connected canonical correlations with tests of subspace independence and with the distribution theory of latent roots.

Sample estimation

For a centered data matrix (\mathbf X\in\mathbb R^{n\times p}) and a centered data matrix (\mathbf Y\in\mathbb R^{n\times q}), the sample covariance blocks are proportional to

[ \mathbf S_{XX}=\mathbf X^{\mathsf T}\mathbf X, \qquad \mathbf S_{YY}=\mathbf Y^{\mathsf T}\mathbf Y, \qquad \mathbf S_{XY}=\mathbf X^{\mathsf T}\mathbf Y. ]

The common normalization factor cancels from the canonical-correlation equations. When (\mathbf S_{XX}) and (\mathbf S_{YY}) are positive definite, the sample canonical correlations are the singular values of

[ \mathbf S_{XX}^{-1/2} \mathbf S_{XY} \mathbf S_{YY}^{-1/2}. ]

The fitted canonical scores are

[ \mathbf U=\mathbf X\mathbf A, \qquad \mathbf V=\mathbf Y\mathbf B. ]

Their numerical values depend on the selected normalization and on arbitrary sign choices for corresponding coefficient vectors. Reversing the signs of both members of a canonical pair leaves its correlation unchanged. Repeated canonical correlations also permit orthogonal rotations within the associated equal-root subspace, so the individual coefficient vectors are not uniquely determined in that case.

When either variable set contains exact linear dependencies, the ordinary inverse-based equations are undefined. A formulation based on the Moore–Penrose inverse restricts the analysis to the observable covariance subspaces. In high-dimensional samples, near-singular covariance estimates can additionally produce unstable coefficient vectors even when the resulting sample correlations remain large.

Interpretation

A canonical correlation measures the strength of association between two optimized linear composites. It does not, by itself, identify which original variables account for that association. The coefficient vectors describe how the canonical scores are constructed, but their magnitudes are affected by correlations among variables within the same set.

Canonical loadings are correlations between the original variables and their own canonical variates. Cross-loadings are correlations between variables in one set and canonical variates from the other set. These quantities describe different aspects of the fitted covariance structure and therefore need not rank variables in the same order as the raw canonical coefficients.

The squared canonical correlation gives the proportion of variance in either standardized canonical variate linearly shared with its paired variate. It does not equal the proportion of total variance in either original variable set explained by the opposite set. Redundancy analysis supplements the canonical correlation by combining the shared variance of a pair with the variance extracted from the original variables.

CCA is invariant under nonsingular linear transformations applied separately to the two variable sets. A change of units, an invertible rotation, or any other invertible reparameterization changes the coefficient vectors but leaves the canonical correlations unchanged. This invariance places the primary interpretation on relationships between the two generated subspaces rather than on a unique coordinate representation.

Relation to regression

When the second set contains a single variable, CCA reduces to the multiple correlation between that variable and the first set. The squared canonical correlation then equals the coefficient of determination from an ordinary least-squares regression with an intercept after centering.

With multiple variables in both sets, CCA can also be understood through reciprocal prediction. The squared canonical correlations are the eigenvalues of the composition of the linear prediction operator from (\mathbf X) to (\mathbf Y) and the corresponding operator from (\mathbf Y) back to (\mathbf X). This relationship distinguishes CCA from fitting a collection of separate regressions, because CCA imposes paired latent directions and treats the two variable sets symmetrically.

Partial least squares regression uses a related paired-score construction but optimizes covariance rather than standardized correlation. As a result, partial least squares depends on marginal scale in a manner that ordinary CCA does not.

Statistical inference

Under multivariate normality, hypotheses concerning the remaining canonical correlations can be expressed through Wilks' lambda. For the hypothesis that canonical correlations (\rho_k,\ldots,\rho_m) vanish, where (m=\min(p,q)), the population likelihood-ratio quantity is represented by

[ \Lambda_k=\prod_{j=k}^{m}(1-\rho_j^2). ]

Its sample analogue substitutes estimated canonical correlations. Bartlett’s transformation gives an asymptotic chi-squared approximation under fixed dimension and increasing sample size. Other latent-root statistics emphasize the largest remaining root or the sum of transformed roots, producing distinct finite-sample behavior while testing related restrictions on the cross-covariance structure.

Classical significance tests concern the population canonical roots rather than the stability of particular coefficient vectors. Sampling variability can be substantial when predictor dimensions approach the sample size, when canonical roots are closely spaced, or when within-set variables are strongly collinear. Resampling distributions therefore characterize a different inferential question from the likelihood-ratio tests.

Regularized forms

Regularized CCA replaces the empirical within-set covariance matrices by stabilized matrices such as

[ \mathbf S_{XX}+\lambda_X\mathbf I ]

and

[ \mathbf S_{YY}+\lambda_Y\mathbf I, ]

where the nonnegative parameters control the contribution of isotropic shrinkage. The resulting canonical correlations depend on the selected regularization and lose the full invariance of ordinary CCA under arbitrary nonsingular transformations. They retain invariance under transformations compatible with the chosen penalty.

Kernel canonical correlation analysis carries the same optimization into reproducing kernel Hilbert spaces. In finite samples, its unregularized form can attain perfect correlations through interpolation, so its mathematical definition commonly includes penalties on the feature-space norms.