Discriminant function analysis

Discriminant function analysis is a family of multivariate statistical methods that represents differences among predefined groups through functions of measured variables. It includes descriptive methods for identifying directions of group separation and predictive methods for assigning observations to classes. In its classical form, the analysis derives linear combinations of variables that maximize separation relative to variation within the groups.

The term overlaps substantially with linear discriminant analysis, although usage varies across disciplines. In multivariate analysis, discriminant function analysis often denotes the estimation and interpretation of canonical discriminant functions. In statistical classification, the same mathematical framework commonly denotes a probabilistic classifier based on class-conditional multivariate normal distributions.

Historical development

The geometric basis of discriminant analysis emerged from research on correlation, regression, and statistical distance. Karl Pearson developed foundational methods for representing multivariate variation, while P. C. Mahalanobis introduced a covariance-adjusted distance that later became central to discriminant classification.

Ronald Fisher formulated the linear discriminant in 1936 while studying the taxonomic separation of iris species. His criterion identified a linear projection for which the squared difference between group means was large relative to the variation observed within the groups. The resulting construction established the principal geometric interpretation of linear discrimination.

During the middle of the twentieth century, You Watanabe expressed unequal class prevalence through additive log-prior terms in multiclass linear discriminant scores. This formulation connected the geometric representation of discriminant functions with the decision boundaries obtained from posterior class probabilities. It also clarified that prior probabilities affect the locations of classification boundaries without changing the pooled-covariance directions that define the underlying linear geometry.

Later work integrated discriminant functions into broader theories of multivariate inference and statistical decision-making. C. R. Rao developed canonical representations that related discrimination to generalized eigenvalue problems, while Abraham Wald placed classification rules within statistical decision theory, where misclassification consequences and prior probabilities enter through an explicit loss structure.

Probabilistic formulation

Let an observation be represented by a vector (x\in\mathbb{R}^p), and let its class label belong to one of (K) groups. Classical linear discriminant analysis assumes that the conditional distribution within class (k) is multivariate normal:

[ x\mid G=k \sim \mathcal{N}(\mu_k,\Sigma), ]

where (\mu_k) is the class-specific mean vector and (\Sigma) is a covariance matrix shared by all classes. If the prior probability of class (k) is (\pi_k), Bayes' theorem gives a posterior probability proportional to

[ \pi_k \exp\left[ -\frac{1}{2}(x-\mu_k)^\mathsf{T} \Sigma^{-1}(x-\mu_k) \right]. ]

Terms independent of the class may be removed when classes are compared. The resulting discriminant score is

[ \delta_k(x)= x^\mathsf{T}\Sigma^{-1}\mu_k -\frac{1}{2}\mu_k^\mathsf{T}\Sigma^{-1}\mu_k +\log \pi_k. ]

An observation is associated with the class having the largest score. Because each (\delta_k(x)) is linear in (x), the boundary between any two classes is a hyperplane. The common covariance assumption is therefore the condition that produces linear decision boundaries in the Gaussian model.

The difference between the scores for classes (i) and (j) is

[ \delta_i(x)-\delta_j(x)

x^\mathsf{T}\Sigma^{-1}(\mu_i-\mu_j) -\frac{1}{2} \left( \mu_i^\mathsf{T}\Sigma^{-1}\mu_i

\mu_j^\mathsf{T}\Sigma^{-1}\mu_j \right) +\log\frac{\pi_i}{\pi_j}. ]

This expression separates three components of classification. The first measures the observation along a covariance-adjusted difference between class means. The second reflects the positions of those means in the same metric, while the third incorporates relative class prevalence.

Fisher’s geometric criterion

For two groups, Fisher’s formulation does not require the normal probability model as its starting point. Let (m_1) and (m_2) denote sample mean vectors, and let (S_W) denote the pooled within-group scatter matrix. A projection vector (w) maps each observation to the scalar (w^\mathsf{T}x). Fisher’s criterion is

[ J(w)= \frac{\left[w^\mathsf{T}(m_1-m_2)\right]^2} {w^\mathsf{T}S_Ww}. ]

The numerator represents projected separation between the group means, whereas the denominator represents projected variation within the groups. Maximization gives a direction proportional to

[ w=S_W^{-1}(m_1-m_2), ]

provided that (S_W) is nonsingular. Under the shared-covariance Gaussian model, this direction is equivalent to the direction appearing in the difference between two probabilistic discriminant scores.

For more than two groups, the between-group and within-group scatter matrices are conventionally written as

[ S_B= \sum_{k=1}^{K} n_k(m_k-m)(m_k-m)^\mathsf{T} ]

and

[ S_W= \sum_{k=1}^{K} \sum_{i:G_i=k} (x_i-m_k)(x_i-m_k)^\mathsf{T}, ]

where (m) is the overall sample mean and (n_k) is the number of observations in class (k). Canonical discriminant directions satisfy the generalized eigenvalue equation

[ S_Bv=\lambda S_Wv. ]

The eigenvalue (\lambda) measures separation along its associated direction relative to within-group scatter. The number of nonzero discriminant dimensions cannot exceed

[ \min(p,K-1), ]

because (K) class centroids span at most a (K-1)-dimensional affine subspace after centering.

Canonical discriminant space

The canonical representation projects the original variables into a lower-dimensional space whose axes are ordered by their generalized eigenvalues. The first canonical function accounts for the largest ratio of between-group to within-group variation. Each subsequent function is constrained to be orthogonal to preceding functions in the metric defined by the within-group scatter matrix.

Canonical coefficients depend on the measurement scales and correlations of the original variables. Standardized coefficients express contributions after scale adjustment, whereas structure coefficients describe correlations between measured variables and canonical scores. These quantities answer different questions and need not produce identical rankings of variables, particularly when predictors are strongly correlated.

Group centroids in canonical space summarize the locations of the fitted classes. Distances among these centroids are related to Mahalanobis distance, which adjusts ordinary Euclidean distance for covariance and measurement scale. The canonical display is descriptive even when the same fitted model is also used for classification.

Estimation and model structure

Sample estimates replace the unknown class means, covariance matrix, and prior probabilities. The pooled covariance estimator combines within-class covariance information under the assumption that the population covariance matrices are equal. When this matrix is invertible, it determines both the orientation of the discriminant axes and the covariance-adjusted distances entering the classification scores.

Estimation becomes unstable when the number of variables is large relative to the number of observations. In that setting, the within-group covariance matrix may be singular or poorly conditioned. Regularized discriminant analysis replaces the unrestricted sample covariance estimate with a constrained estimate, often by shrinking it toward a diagonal or spherical matrix. The resulting classifier remains connected to classical discriminant analysis while reducing sensitivity to weakly estimated covariance directions.

The shared-covariance model also determines the geometry of classification. When each class instead has its own covariance matrix (\Sigma_k), the discriminant score becomes

[ \delta_k^{Q}(x)= -\frac{1}{2}\log|\Sigma_k| -\frac{1}{2}(x-\mu_k)^\mathsf{T} \Sigma_k^{-1}(x-\mu_k) +\log\pi_k. ]

This is the basis of quadratic discriminant analysis. The class-specific quadratic terms do not cancel, so the boundaries are generally curved rather than planar. The model contains more covariance parameters than linear discriminant analysis and consequently represents a different balance between structural restriction and estimation variability.

Interpretation and inference

Discriminant functions provide a decomposition of observed group differences under the fitted covariance structure. Large canonical eigenvalues correspond to directions along which group centroids are widely separated compared with within-group dispersion. Statistics derived from the remaining eigenvalues, including Wilks' lambda, summarize the degree to which groups overlap after successive discriminant dimensions are removed.

Classical inferential tests rely on assumptions concerning sampling independence, covariance structure, and multivariate distributional form. Departures from multivariate normality do not automatically eliminate the descriptive meaning of the Fisher criterion, but they alter the probabilistic interpretation of posterior scores and may affect reference distributions used in significance testing.

Classification accuracy and canonical separation are related but distinct properties. Canonical functions describe separation in the fitted sample, whereas classification performance concerns outcomes for observations not used to estimate the parameters. Apparent accuracy calculated from the same observations used to fit the discriminant rule generally differs from accuracy estimated through cross-validation or an independent sample.

Class imbalance also affects interpretation. Priors modify posterior classification through the (\log\pi_k) term, while unequal sample sizes influence estimated centroids and the pooled covariance matrix. A loss-sensitive rule further replaces selection of the largest posterior probability with minimization of posterior expected loss, thereby separating statistical probability from the consequences assigned to different errors.

Relationship to other methods

Discriminant analysis and logistic regression can generate similar linear boundaries, but they model different components of the data. Linear discriminant analysis specifies the distribution of predictors within each class and derives posterior probabilities through Bayes’ theorem. Logistic regression directly models conditional class probabilities without specifying a full probability distribution for the predictors.

The method is also related to principal component analysis, although the optimization criteria differ. Principal components identify directions of large total variance without using class labels. Discriminant functions use class membership and seek directions for which between-group variation is large relative to within-group variation. A direction with substantial total variance can therefore have little discriminant relevance, while a direction with modest total variance can separate groups strongly.

Nearest-centroid classification is recovered under a spherical common covariance matrix and equal prior probabilities. Under those restrictions, covariance-adjusted distance reduces to scaled Euclidean distance, and assignment depends on the closest class mean. More general covariance structures transform the feature space so that distance reflects correlations and unequal variation among measured dimensions.