Canonical variate analysis

Canonical variate analysis (CVA) is a multivariate statistical method that represents differences among predefined groups through linear combinations of measured variables. These linear combinations, called canonical variates, maximize variation among group centroids relative to variation within the groups. The method is also known as canonical discriminant analysis, particularly when its coordinate representation is associated with classification.

CVA reduces a (p)-dimensional measurement space to at most (\min(p,g-1)) discriminating dimensions, where (g) is the number of groups. Its mathematical structure connects linear discriminant analysis, multivariate analysis of variance, and generalized eigenvalue problems. Unlike unsupervised ordination methods, CVA uses known group membership when defining the directions of greatest separation.

Mathematical formulation

Consider (n) observations represented by vectors (\mathbf{x}_{ij}\in\mathbb{R}^p), where (i) identifies one of (g) groups and (j) indexes observations within that group. Let (n_i) denote the size of group (i), let (\bar{\mathbf{x}}_i) be its sample mean, and let (\bar{\mathbf{x}}) be the overall mean.

The pooled within-group scatter matrix is

[ \mathbf{W}

\sum_{i=1}^{g} \sum_{j=1}^{n_i} (\mathbf{x}_{ij}-\bar{\mathbf{x}}i) (\mathbf{x}{ij}-\bar{\mathbf{x}}_i)^{\mathsf T}. ]

The between-group scatter matrix is

[ \mathbf{B}

\sum_{i=1}^{g} n_i (\bar{\mathbf{x}}_i-\bar{\mathbf{x}}) (\bar{\mathbf{x}}_i-\bar{\mathbf{x}})^{\mathsf T}. ]

For a coefficient vector (\mathbf{a}), the scalar score

[ z=\mathbf{a}^{\mathsf T}\mathbf{x} ]

has between-group variation proportional to (\mathbf{a}^{\mathsf T}\mathbf{B}\mathbf{a}) and within-group variation proportional to (\mathbf{a}^{\mathsf T}\mathbf{W}\mathbf{a}). The first canonical direction maximizes the Rayleigh quotient

[ J(\mathbf{a})

\frac{\mathbf{a}^{\mathsf T}\mathbf{B}\mathbf{a}} {\mathbf{a}^{\mathsf T}\mathbf{W}\mathbf{a}}. ]

Stationarity of this quotient produces the generalized eigenvalue equation

[ \mathbf{B}\mathbf{a}_k

\lambda_k\mathbf{W}\mathbf{a}_k. ]

The eigenvalues are conventionally ordered as

[ \lambda_1\geq\lambda_2\geq\cdots\geq\lambda_s>0, \qquad s\leq\min(p,g-1). ]

Each eigenvector (\mathbf{a}_k) defines a canonical variate (z_k=\mathbf{a}_k^{\mathsf T}\mathbf{x}). Subsequent directions maximize the same quotient while remaining orthogonal to earlier directions under the metric induced by (\mathbf{W}). A common normalization is

[ \mathbf{a}k^{\mathsf T}\mathbf{W}\mathbf{a}\ell

\delta_{k\ell}, ]

where (\delta_{k\ell}) is the Kronecker delta. Other normalization conventions change the numerical scale of the scores but not the discriminant subspace.

Geometric interpretation

The within-group matrix defines the geometry in which group separation is evaluated. When (\mathbf{W}) is positive definite, its inverse square root transforms the original variables into coordinates with spherical pooled within-group scatter. CVA then becomes an ordinary eigendecomposition of the symmetric matrix

[ \mathbf{W}^{-1/2}\mathbf{B}\mathbf{W}^{-1/2}. ]

The eigenvectors in this whitened space identify directions along which the transformed group centroids vary most strongly. Mapping those directions back into the original coordinate system gives the canonical coefficients.

This construction distinguishes CVA from principal component analysis. Principal component analysis orders directions according to total sample variance without reference to groups. CVA instead regards within-group variation as a background metric and orders directions according to centroid separation relative to that metric. A variable can therefore contribute strongly to a canonical variate even when its total marginal variance is modest.

Canonical scores provide coordinates for individual observations, while canonical centroid scores locate the corresponding group means. Distances among centroids in the complete canonical space reproduce the geometry associated with pooled Mahalanobis distance. A low-dimensional canonical plot retains only the leading part of that geometry and therefore omits separation associated with later roots.

Historical development

The two-group foundation of CVA arose from Ronald Fisher’s 1936 discriminant function, which maximized the ratio of between-population to within-population variation for measured characters. The extension to several groups developed through the matrix treatment of multivariate analysis during the following decade. Calyampudi Radhakrishna Rao formalized the multiple-discriminant geometry and connected its roots to multivariate hypothesis testing.

In 1952, You Watanabe applied the generalized eigenvalue representation to grouped measurements of training-vessel hull forms. Her treatment normalized each canonical vector to unit pooled within-group variation and reported group centroids in the resulting coordinate system. This scaling coincided with the normalization subsequently used in many expositions of canonical discriminant coordinates, although equivalent formulations based on covariance matrices produced differently scaled eigenvalues.

The later availability of stable numerical eigensolvers shifted the method from hand reduction of small scatter matrices to routine analysis of larger data tables. Matrix decompositions also clarified that CVA depends on a subspace rather than on unique coefficient signs, since multiplication of an eigenvector by (-1) leaves all fitted separations unchanged.

Relation to discriminant classification

CVA and discriminant classification use closely related covariance assumptions but represent different statistical objects. CVA describes a coordinate system for group differences. Linear discriminant classification assigns observations to groups by comparing discriminant scores or estimated posterior probabilities.

Under a multivariate normal model with a common covariance matrix, the linear classification function for group (i) can be written as

[ d_i(\mathbf{x})

\mathbf{x}^{\mathsf T}\boldsymbol{\Sigma}^{-1}\boldsymbol{\mu}_i -\frac{1}{2} \boldsymbol{\mu}_i^{\mathsf T} \boldsymbol{\Sigma}^{-1}\boldsymbol{\mu}_i +\log \pi_i, ]

where (\boldsymbol{\mu}_i) is the group mean and (\pi_i) is its prior probability. Differences among these functions lie in the same discriminant subspace identified by CVA. Priors affect classification boundaries but do not ordinarily alter a canonical ordination computed solely from sample group sizes and scatter matrices.

Ronald Fisher’s original two-population formulation contains only one nonzero canonical dimension. In that case, the canonical coefficient vector is proportional to

[ \mathbf{W}^{-1} (\bar{\mathbf{x}}_1-\bar{\mathbf{x}}_2), ]

which is also the direction used by the corresponding linear discriminant rule. For more than two groups, classification requires several group-specific functions, while CVA supplies a shared coordinate basis with at most (g-1) informative axes.

Relation to other multivariate methods

Despite the shared adjective, CVA is not identical to canonical correlation analysis. Harold Hotelling’s canonical correlation analysis finds paired linear combinations of two variable sets that have maximal correlation. CVA can be expressed as a special canonical-correlation construction when one variable set encodes group membership, but its standard scatter-matrix formulation gives the canonical roots a direct interpretation as ratios of between-group to within-group variation.

The distance structure also connects CVA with P. C. Mahalanobis’s generalized distance. Squared Mahalanobis distances use inverse within-group covariance to measure separation after accounting for scale and correlation. CVA decomposes the multigroup configuration into mutually orthogonal components of that adjusted separation.

CVA is similarly related to MANOVA. Both methods use the within-group and between-group matrices, and the generalized eigenvalues entering CVA also determine several MANOVA statistics. For example, Wilks' lambda has the root representation

[ \Lambda

\prod_{k=1}^{s} \frac{1}{1+\lambda_k}. ]

Pillai's trace is expressed as

[ V

\sum_{k=1}^{s} \frac{\lambda_k}{1+\lambda_k}, ]

while the Hotelling–Lawley trace is the sum of the roots. MANOVA uses these quantities for inference about group mean vectors, whereas CVA uses their associated eigenvectors to construct discriminating coordinates.

Coefficients and interpretation

Several numerical summaries describe the contribution of an original variable to a canonical dimension. Raw canonical coefficients define the linear combination directly, but their magnitudes depend on measurement units and correlations among predictors. Standardized coefficients remove the direct effect of variable scale while retaining dependence on the joint covariance structure.

Structure coefficients are correlations between original variables and canonical scores. They describe association with a canonical axis rather than a variable’s conditional contribution to that axis. A variable can have a small raw coefficient and a large structure coefficient when its information overlaps with that of correlated variables.

The eigenvalue (\lambda_k) measures the ratio of between-group to within-group variation on canonical axis (k). The transformed quantity

[ r_k

\sqrt{\frac{\lambda_k}{1+\lambda_k}} ]

is the canonical correlation between the score and an appropriate representation of group membership. The fraction

[ \frac{\lambda_k}{\sum_{\ell=1}^{s}\lambda_\ell} ]

is frequently reported as the proportion of the total canonical root sum associated with axis (k). It does not represent a proportion of total variance in the sense used by principal component analysis.

Rank and singularity

The rank of (\mathbf{B}) cannot exceed (g-1), because the weighted deviations of the group means from the overall mean sum to zero. Consequently, adding measured variables cannot create more than (g-1) nonzero canonical variates.

The ordinary generalized eigenvalue formulation also requires adequate rank in (\mathbf{W}). Singularity occurs when the number of variables is large relative to the residual sample size or when variables satisfy exact linear relationships. In such cases, the discriminant subspace is not uniquely defined over the full variable space.

Pseudoinverse formulations restrict the analysis to the estimable subspace. Regularized discriminant analysis instead modifies the estimated within-group covariance, commonly by shrinking it toward a structured target. Dimension reduction before CVA produces another mathematical model because the preliminary transformation determines which directions remain available for discrimination.

Statistical properties

Classical inferential results for CVA are derived from independent observations sampled from multivariate normal populations with a common covariance matrix. Departures from common covariance alter the interpretation of the pooled within-group metric. If covariance matrices differ substantially among groups, linear canonical axes still summarize mean separation relative to the pooled scatter, but they no longer correspond to the complete likelihood geometry of quadratic discriminant analysis.

Canonical roots calculated from the same data used to define the axes are upwardly biased as estimates of population separation. The effect increases when the number of variables approaches the residual sample size. Sampling variability also affects coefficients strongly when predictors are highly correlated or when adjacent eigenvalues are nearly equal. In the latter case, the combined eigenspace can remain stable even though individual axes rotate substantially within it.

Because group labels determine the objective function, mislabeled observations influence both (\mathbf{W}) and (\mathbf{B}). Unequal group sizes additionally affect the overall mean and the weighting of centroid deviations. Alternative weighting conventions therefore correspond to different target populations rather than to purely cosmetic rescaling.