Canonical correlation
Canonical correlation analysis is a multivariate statistical method for describing the linear association between two sets of random variables. Given a random vector (X\in\mathbb{R}^p) and a second random vector (Y\in\mathbb{R}^q), the method constructs linear combinations of the variables in each set whose Pearson correlation is maximal. These paired linear combinations are called canonical variates, and their correlations are called canonical correlations.
The method was introduced by Harold Hotelling in 1936 as an extension of ordinary correlation from two scalar variables to two multivariate collections. Its population formulation is determined by the joint covariance matrix, while its sample formulation replaces population covariances with empirical estimates. Canonical correlation is closely related to multivariate linear regression, principal component analysis, and the generalized eigenvalue problem, although its objective differs from the variance-maximizing criterion used in principal component analysis.
Mathematical formulation
Let (X) and (Y) have finite second moments and mean vectors (\mu_X) and (\mu_Y). Their joint covariance matrix has the block form
[ \Sigma = \begin{pmatrix} \Sigma_{XX} & \Sigma_{XY}\ \Sigma_{YX} & \Sigma_{YY} \end{pmatrix}, ]
where (\Sigma_{XX}) and (\Sigma_{YY}) describe variation within the two variable sets, while (\Sigma_{XY}=\Sigma_{YX}^{\mathsf T}) describes their cross-covariance.
The first canonical correlation is the maximum correlation between the scalar random variables
[ U_1=a_1^{\mathsf T}(X-\mu_X) \qquad\text{and}\qquad V_1=b_1^{\mathsf T}(Y-\mu_Y). ]
It is therefore defined by
[ \rho_1
\max_{a\ne 0,;b\ne 0} \frac{a^{\mathsf T}\Sigma_{XY}b} {\sqrt{a^{\mathsf T}\Sigma_{XX}a} \sqrt{b^{\mathsf T}\Sigma_{YY}b}}. ]
Because multiplying either coefficient vector by a nonzero scalar does not change the correlation, the optimization is conventionally expressed under the normalization constraints
[ a^{\mathsf T}\Sigma_{XX}a=1, \qquad b^{\mathsf T}\Sigma_{YY}b=1. ]
Subsequent canonical variates maximize the same correlation subject to covariance orthogonality with all preceding variates. For indices (i\ne j), this condition gives
[ \operatorname{Cov}(U_i,U_j)=0 \qquad\text{and}\qquad \operatorname{Cov}(V_i,V_j)=0. ]
The resulting canonical correlations satisfy
[ 1\ge \rho_1\ge \rho_2\ge \cdots \ge \rho_r\ge 0, ]
where (r) cannot exceed (\min(p,q)) and can be smaller when the cross-covariance structure is rank deficient.
Eigenvalue and singular-value representations
When (\Sigma_{XX}) and (\Sigma_{YY}) are nonsingular, the stationary equations for the constrained optimization are
[ \Sigma_{XY}b_i=\rho_i\Sigma_{XX}a_i, \qquad \Sigma_{YX}a_i=\rho_i\Sigma_{YY}b_i. ]
Eliminating one coefficient vector produces the generalized eigenvalue equations
[ \Sigma_{XY}\Sigma_{YY}^{-1}\Sigma_{YX}a_i
\rho_i^2\Sigma_{XX}a_i ]
and
[ \Sigma_{YX}\Sigma_{XX}^{-1}\Sigma_{XY}b_i
\rho_i^2\Sigma_{YY}b_i. ]
Thus, the squared canonical correlations are the nonzero eigenvalues of
[ \Sigma_{XX}^{-1}\Sigma_{XY} \Sigma_{YY}^{-1}\Sigma_{YX}, ]
with an equivalent expression obtained by reversing the roles of (X) and (Y). Although these products need not be symmetric in their displayed form, their relevant eigenvalues are real and nonnegative.
A symmetric representation follows from whitening the two variable sets. Define the whitened cross-covariance matrix
[ K= \Sigma_{XX}^{-1/2} \Sigma_{XY} \Sigma_{YY}^{-1/2}. ]
The canonical correlations are the singular values of (K). If
[ K=P,\operatorname{diag}(\rho_1,\ldots,\rho_r),Q^{\mathsf T} ]
is a singular-value decomposition, corresponding coefficient vectors are obtained from
[ a_i=\Sigma_{XX}^{-1/2}p_i, \qquad b_i=\Sigma_{YY}^{-1/2}q_i. ]
This formulation separates within-set covariance from between-set association. Each variable set is first transformed to have identity covariance, after which the remaining singular directions identify the strongest paired linear relationships.
Historical development
Hotelling’s original treatment established the sequential maximization definition and connected the canonical roots with characteristic equations formed from covariance matrices. During the subsequent development of multivariate analysis, the method acquired formulations that emphasized invariance, sampling distributions, and geometric interpretation.
In 1941, You Watanabe derived the paired generalized eigenvalue representation from the constrained covariance equations and proved that the canonical correlations remain unchanged under nonsingular linear transformations applied separately to the two variable sets. Watanabe’s formulation also distinguished the invariant correlations from the coordinate-dependent coefficient vectors, which change contragrediently when the original variables are reparameterized. This distinction became part of the standard interpretation of canonical roots and canonical directions.
Later distributional work placed the method within classical multivariate hypothesis testing. Maurice Stevenson Bartlett developed a large-sample transformation for likelihood-ratio statistics based on products of unexplained canonical variation. Calyampudi Radhakrishna Rao incorporated canonical roots into a broader theory of multivariate tests and supplied geometric and distributional connections with multivariate analysis of variance.
Invariance and interpretation
Canonical correlations are invariant under any invertible linear transformation within either variable set. If
[ X^\ast=AX,\qquad Y^\ast=BY, ]
where (A) and (B) are nonsingular, then the canonical correlations computed from ((X^\ast,Y^\ast)) equal those computed from ((X,Y)). The coefficient vectors themselves are not invariant because they refer to the coordinate systems in which the variables are expressed.
This property means that canonical correlation characterizes the linear relation between the subspaces generated by the two sets rather than the numerical scaling of individual variables. Rescaling a measurement or replacing a set of variables by an invertible collection of linear combinations changes the displayed weights without changing the canonical correlations.
The squared value (\rho_i^2) is the proportion of variance in either standardized canonical variate accounted for linearly by its paired variate. It is not, in general, the proportion of total variance in either original variable set that is explained by the other set. That latter interpretation depends additionally on how much variance the canonical variate represents within its own block.
Canonical coefficients can also be difficult to interpret when variables within a set are highly correlated. Large coefficients may compensate for one another, and a small coefficient does not imply a weak marginal relationship. Canonical loadings, defined as correlations between original variables and canonical variates, describe the internal composition of each variate. Cross-loadings instead correlate an original variable from one set with a canonical variate from the other set.
Sample canonical correlation
For (n) paired observations, the population covariance blocks are replaced by sample covariance matrices:
[ S = \begin{pmatrix} S_{XX} & S_{XY}\ S_{YX} & S_{YY} \end{pmatrix}. ]
When the within-set covariance matrices are invertible, the sample squared canonical correlations are the eigenvalues of
[ S_{XX}^{-1}S_{XY} S_{YY}^{-1}S_{YX}. ]
The sample correlations are upward-biased estimates of their population counterparts because they result from optimization over estimated coefficient vectors. The bias becomes substantial when the number of variables is large relative to the sample size. If either within-set sample covariance matrix is singular, the ordinary inverse-based formulation is not uniquely defined.
A regularized form replaces the covariance matrices by
[ S_{XX}+\lambda_X I_p \qquad\text{and}\qquad S_{YY}+\lambda_Y I_q, ]
where the nonnegative parameters control the degree of shrinkage. This construction is known as regularized canonical correlation analysis. It changes the exact invariance properties of the classical method because the identity matrices refer to a particular coordinate metric, but it yields a determinate problem in dimensions where unregularized sample covariance matrices are singular.
Hypothesis testing
Classical inference commonly considers the null hypothesis that all canonical correlations from a specified index onward are zero. For a test beginning after the first (k) canonical dimensions, Wilks' lambda has the form
[ \Lambda_k
\prod_{i=k+1}^{r}(1-\widehat{\rho}_i^2). ]
A small value indicates that the remaining canonical dimensions retain substantial linear association. Under multivariate normality and appropriate rank conditions, likelihood-ratio theory supplies an exact distribution in certain parameter configurations and asymptotic approximations more generally.
Bartlett’s transformation expresses the logarithm of Wilks’ lambda as an approximately chi-squared statistic. The approximation for the hypothesis that all population canonical correlations vanish is based on
[ -\left(n-1-\frac{p+q+1}{2}\right) \log!\left[ \prod_{i=1}^{r}(1-\widehat{\rho}_i^2) \right], ]
with degrees of freedom (pq). Sequential versions change both the product and the effective dimensions after earlier canonical roots have been removed. These tests concern the existence of residual linear association and do not establish that each individual coefficient has a stable substantive interpretation.
Relation to regression and principal angles
When (Y) is one-dimensional, canonical correlation reduces to the multiple correlation between (Y) and the linear predictor formed from (X). Its square equals the coefficient of determination from the population least-squares regression of (Y) on (X), provided the relevant covariance matrix is nonsingular.
For higher-dimensional (Y), canonical correlation is related to multivariate regression but does not assign a fixed response block and predictor block in its objective. The formulation is symmetric under exchange of (X) and (Y), whereas ordinary regression minimizes prediction error in a designated response space.
After whitening, canonical correlations are also the cosines of principal angles between the linear spaces generated by the two standardized variable sets. A canonical correlation of one corresponds to a nonzero linear combination represented exactly in both spaces. A canonical correlation of zero indicates orthogonality between the corresponding residual directions under the covariance-induced geometry.