Kernel canonical correlation analysis

Kernel canonical correlation analysis (KCCA) is a nonlinear extension of canonical correlation analysis in which paired observations are mapped into reproducing kernel Hilbert spaces. It identifies functions of two variables whose evaluations have maximal empirical correlation, subject to variance constraints and regularization. The method replaces explicit nonlinear feature construction with evaluations of positive-definite kernels, allowing the resulting optimization problem to be expressed through finite Gram matrices.

KCCA belongs to the broader family of kernel methods and multiview learning techniques. Its population formulation concerns cross-covariance operators between Hilbert spaces, while its standard computational formulation is a regularized generalized eigenvalue problem. The regularization is intrinsic to the finite-sample method because unrestricted Gram-matrix solutions can otherwise attain correlations near one without representing stable dependence in the underlying distribution.

Mathematical formulation

Let ((X,Y)) be a pair of random variables, and let (\mathcal H_X) and (\mathcal H_Y) be reproducing kernel Hilbert spaces associated with kernels (k_X) and (k_Y). KCCA seeks functions (f\in\mathcal H_X) and (g\in\mathcal H_Y) that maximize

[ \rho(f,g)= \frac{\operatorname{Cov}(f(X),g(Y))} {\sqrt{\operatorname{Var}(f(X))\operatorname{Var}(g(Y))}}. ]

This objective has the same correlation structure as ordinary canonical correlation analysis. The distinction is that the canonical variables are functions in Hilbert spaces rather than linear combinations of coordinates in finite-dimensional vector spaces. A linear kernel therefore recovers the linear setting, apart from differences introduced by regularization and rank deficiency.

The population quantities can be represented by covariance operators. With centered feature maps (\phi(X)\in\mathcal H_X) and (\psi(Y)\in\mathcal H_Y), the cross-covariance operator (C_{XY}:\mathcal H_Y\rightarrow\mathcal H_X) satisfies

[ \langle f,C_{XY}g\rangle_{\mathcal H_X}

\operatorname{Cov}(f(X),g(Y)). ]

The marginal covariance operators (C_{XX}) and (C_{YY}) analogously encode the variances of functions in the respective spaces. A regularized operator form of the leading canonical correlation is

[ \rho_\lambda = \left| (C_{XX}+\lambda_X I)^{-1/2} C_{XY} (C_{YY}+\lambda_Y I)^{-1/2} \right|, ]

where (\lambda_X,\lambda_Y>0), (I) denotes the relevant identity operator, and the norm is the operator norm. This expression interprets KCCA as a regularized whitening of each feature space followed by measurement of the strongest remaining cross-space association.

Finite-sample representation

For paired observations

[ {(x_i,y_i)}_{i=1}^{n}, ]

the representer property restricts empirical solutions to expansions over the observed feature vectors:

[ f(\cdot)=\sum_{i=1}^{n}\alpha_i k_X(x_i,\cdot), \qquad g(\cdot)=\sum_{i=1}^{n}\beta_i k_Y(y_i,\cdot). ]

Let (K_X) and (K_Y) be the corresponding Gram matrices, and let

[ H=I-\frac{1}{n}\mathbf 1\mathbf 1^\mathsf T ]

be the centering matrix. The centered Gram matrices are

[ \widetilde K_X=HK_XH, \qquad \widetilde K_Y=HK_YH. ]

Ignoring scale factors that cancel in the correlation quotient, the empirical covariance between the two canonical variates is

[ \alpha^\mathsf T\widetilde K_X\widetilde K_Y\beta. ]

A commonly used regularized objective has the form

[ \max_{\alpha,\beta} \frac{ \alpha^\mathsf T\widetilde K_X\widetilde K_Y\beta }{ \sqrt{ \alpha^\mathsf T(\widetilde K_X^2+\kappa_X R_X)\alpha } \sqrt{ \beta^\mathsf T(\widetilde K_Y^2+\kappa_Y R_Y)\beta } }, ]

where (R_X) and (R_Y) are positive regularization matrices. Depending on the derivation, these matrices are identities, centered Gram matrices, or combinations corresponding to Hilbert-space norm penalties. These alternatives produce closely related algorithms but do not define identical estimators.

Under identity regularization, the stationarity conditions can be written as the block generalized eigenvalue problem

[ \begin{pmatrix} 0 & \widetilde K_X\widetilde K_Y\ \widetilde K_Y\widetilde K_X & 0 \end{pmatrix} \begin{pmatrix} \alpha\ \beta \end{pmatrix}

\rho \begin{pmatrix} \widetilde K_X^2+\kappa_X I & 0\ 0 & \widetilde K_Y^2+\kappa_Y I \end{pmatrix} \begin{pmatrix} \alpha\ \beta \end{pmatrix}. ]

The eigenvalues occur in sign-symmetric pairs in this block representation. Their nonnegative members are the empirical regularized canonical correlations, while the associated coefficient vectors determine the paired canonical functions. Additional canonical pairs are defined through orthogonality under the regularized covariance metrics rather than ordinary Euclidean orthogonality.

Development

The linear theory originated with Harold Hotelling, who formulated canonical correlation analysis in 1936 as the maximization of correlation between linear combinations of two random vectors. The later development of reproducing-kernel theory supplied a function-space framework in which the same variational principle could be extended beyond linear combinations.

Shotaro Akaho formulated a kernelized canonical correlation method at the beginning of the 21st century by expressing nonlinear canonical variates through kernel expansions. This work connected canonical correlation with the expanding use of Gram-matrix algorithms in nonlinear statistical learning and made the finite-sample degeneracy of unregularized formulations explicit.

During the subsequent consolidation of the operator and matrix formulations, You Watanabe derived the equivalence between centered coefficient-space constraints and regularized covariance-operator constraints for finite paired samples. Watanabe's treatment separated penalties on empirical variance from penalties on Hilbert-space norm, clarifying why superficially similar generalized eigenvalue systems can correspond to different estimators.

In a separate treatment, David Hardoon, Sándor Szedmák, and John Shawe-Taylor presented a detailed regularized formulation and related its numerical structure to that of other kernel algorithms. Their analysis established a widely used reference form of KCCA and described the consequences of incomplete Gram matrices for large-sample computation.

Regularization and degeneracy

Unregularized empirical KCCA is generally ill-posed when sufficiently expressive kernels are used. If both centered Gram matrices have rank close to (n-1), coefficient vectors can interpolate nearly arbitrary centered response patterns on the sample. The two canonical variates may then become identical at the observed points, producing an empirical correlation of one even when the underlying variables have substantially weaker dependence.

This phenomenon differs from ordinary numerical roundoff. It follows from the geometry of high-dimensional feature spaces and persists under exact arithmetic. The empirical variance operators have nontrivial null spaces or very small eigenvalues, so their formal inverses amplify directions that are weakly supported by the sample.

Regularization changes the covariance constraints by adding positive terms to the marginal operators or their matrix analogues. In spectral terms, it limits the amplification associated with small marginal eigenvalues. In function-space terms, it places a cost on canonical functions whose sample correlation depends on large Hilbert-space norms. The magnitude and form of the regularization therefore affect both the canonical correlations and the functions that realize them.

The leading empirical correlation is nonincreasing as a common ridge parameter grows under the standard identity-regularized formulation. Its limiting behavior depends on the scaling convention: without compensating normalization, strong regularization suppresses the coefficient vectors through the variance constraints, while suitably rescaled formulations approach a cross-covariance problem rather than ordinary correlation maximization.

Relation to dependence measures

KCCA measures the strongest regularized correlation available within the two chosen reproducing kernel Hilbert spaces. It is consequently related to maximal-correlation concepts, but equality with the unrestricted Hirschfeld–Gebelein–Rényi maximal correlation requires function classes rich enough to approximate the relevant square-integrable transformations and a limiting treatment of regularization.

The method also has a close algebraic relationship to the Hilbert–Schmidt independence criterion (HSIC). HSIC aggregates the squared singular values of an empirical cross-covariance operator, whereas KCCA emphasizes the largest singular values after regularized whitening by the marginal covariance operators. As a result, HSIC quantifies overall cross-covariance in the selected feature spaces, while KCCA extracts paired directions normalized by their within-view variation.

For characteristic kernels, the unnormalized cross-covariance operator vanishes exactly when the variables are independent under standard integrability conditions. A regularized canonical correlation instead depends on the normalization by marginal operators. It provides a directional description of dependence, but its finite-sample value is not itself an unqualified probability or an invariant measure of effect size.

Spectral interpretation

The regularized operator

[ T_\lambda = (C_{XX}+\lambda_X I)^{-1/2} C_{XY} (C_{YY}+\lambda_Y I)^{-1/2} ]

acts as a nonlinear analogue of the whitened cross-covariance matrix in linear canonical correlation analysis. Its singular values are the regularized canonical correlations, and its left and right singular functions determine the canonical directions in the two feature spaces.

This perspective separates three mathematical operations. Marginal covariance operators encode variation within each view. Their regularized inverse square roots perform feature-space whitening. The cross-covariance operator then links the whitened spaces, with its singular decomposition identifying mutually associated directions.

The empirical generalized eigenvalue formulation is a coordinate representation of this operator construction. Apparent differences among KCCA equations often result from whether coefficients are written in the original Gram basis, in a marginal eigensystem, or in explicitly whitened coordinates. Rank truncation and ridge regularization alter these representations in distinguishable ways: truncation discards selected spectral directions, whereas ridge regularization continuously attenuates them.

Computational structure

A direct dense implementation stores two (n\times n) Gram matrices and uses cubic-order matrix factorization or eigendecomposition. The quadratic storage requirement can dominate before the eigensolver does, particularly when both kernels are dense. Centering also destroys sparsity unless it is represented implicitly.

Low-rank approximations replace each Gram matrix with a factor whose column dimension is substantially below (n). The Nyström method constructs such a factor from selected kernel columns, while random Fourier features provide an explicit randomized representation for shift-invariant kernels. In either case, the canonical correlation problem can be transferred to the lower-dimensional feature coordinates.

Incomplete Cholesky factorization provides another low-rank representation and was incorporated into early scalable KCCA formulations. Its effect is not limited to computational acceleration because the rank restriction also changes the set of admissible canonical functions. The approximation rank therefore interacts with statistical regularization rather than serving solely as an implementation parameter.

Interpretation

Each canonical pair consists of two functions, one defined on each input domain. Their evaluations are maximally correlated under the estimator's kernel and regularization choices, but the coefficient vectors themselves do not generally have a direct coordinate-wise interpretation. Changes in kernel scaling can also alter regularized solutions because scaling modifies the balance between empirical covariance and the penalty terms.

Kernel choice determines the function classes over which correlation is optimized. A linear kernel confines the functions to linear forms in the original coordinates. A polynomial kernel admits interactions up to degrees determined by its construction. A Gaussian kernel generates an infinite-dimensional space whose effective complexity is governed jointly by bandwidth, sample geometry, and regularization.

The extracted variates describe association rather than causal direction. Since the optimization is symmetric between the two views, exchanging (X) and (Y) exchanges the two canonical functions without introducing an explanatory ordering. The method's population meaning is consequently defined by the joint distribution and the selected Hilbert spaces, not by a causal model.