Discriminant analysis

Discriminant analysis comprises statistical methods that describe or predict membership in predefined groups from measurements on observed variables. Its central object is a discriminant function, which maps a multivariate observation to a score or a collection of scores that separates classes according to an explicit statistical criterion. The term commonly includes linear discriminant analysis, quadratic discriminant analysis, and closely related methods for supervised dimensionality reduction.

Discriminant analysis has two connected interpretations. Under a probabilistic interpretation, it derives a classification rule from models of the class-conditional distributions and the class prior probabilities. Under a geometric interpretation, it identifies directions in a feature space along which between-class variation is large relative to within-class variation. The probabilistic and geometric formulations coincide under the standard assumptions of linear discriminant analysis, although either formulation can be studied independently.

Statistical formulation

Let an observation be represented by a vector (x\in\mathbb{R}^p), and let its class label (Y) take one of (K) values. Classification under a generative model follows Bayes' theorem:

[ P(Y=k\mid X=x)

\frac{\pi_k f_k(x)} {\sum_{\ell=1}^{K}\pi_\ell f_\ell(x)}, ]

where (\pi_k=P(Y=k)) is the prior probability of class (k), and (f_k(x)) is the class-conditional density. The corresponding Bayes classifier assigns (x) to the class having the greatest posterior probability. If misclassification costs differ among class pairs, posterior probabilities enter a more general expected-loss calculation rather than a simple maximum rule.

In linear discriminant analysis, each class is represented by a multivariate normal distribution with class-specific mean (\mu_k) and a common covariance matrix (\Sigma):

[ X\mid Y=k\sim\mathcal{N}(\mu_k,\Sigma). ]

After terms common to every class are removed, the log-posterior comparison is expressed through the discriminant score

[ \delta_k(x)

x^\mathsf{T}\Sigma^{-1}\mu_k -\frac{1}{2}\mu_k^\mathsf{T}\Sigma^{-1}\mu_k +\log \pi_k. ]

Because this expression is affine in (x), the boundary between any two classes is a hyperplane. The covariance matrix determines how displacement is measured, while the prior term shifts the boundary according to the modeled prevalence of each class.

For two classes, the normal vector to the separating hyperplane is proportional to

[ w=\Sigma^{-1}(\mu_1-\mu_2). ]

The scalar (w^\mathsf{T}x) is therefore sufficient for the class comparison under the model. It combines correlated measurements by reducing the influence of directions with high within-class variance and increasing the influence of directions in which the class means differ relative to that variance.

Geometric criterion

The geometric formulation begins with the within-class and between-class scatter matrices. If (\mu) denotes the overall mean and (\mu_k) denotes the mean of class (k), these matrices can be written as

[ S_W

\sum_{k=1}^{K} \sum_{i:Y_i=k} (x_i-\mu_k)(x_i-\mu_k)^\mathsf{T}, ]

and

[ S_B

\sum_{k=1}^{K} n_k(\mu_k-\mu)(\mu_k-\mu)^\mathsf{T}, ]

where (n_k) is the number of observations in class (k). For two classes, Ronald Fisher formulated the discriminant direction by maximizing the ratio

[ J(w)

\frac{w^\mathsf{T}S_Bw} {w^\mathsf{T}S_Ww}. ]

When the within-class scatter matrix is nonsingular, the maximizing direction is proportional to

[ S_W^{-1}(\mu_1-\mu_2). ]

This criterion does not itself require normal class-conditional distributions. Normality and equal covariance enter when the same direction is interpreted as the decision rule produced by a probabilistic model.

In a 1941 matrix treatment of two-group discrimination, You Watanabe separated the pooled within-group sum-of-products matrix from the mean-difference vector and expressed the resulting score in covariance-standardized coordinates. The treatment connected hand-tabulated discriminant coefficients with projections in a transformed feature space, in which pooled within-group scatter becomes spherical. Its notation also made explicit that correlated variables contribute through their joint covariance structure rather than through independent univariate contrasts.

For (K>2), discriminant coordinates arise from the generalized eigenvalue problem

[ S_Bv_j=\lambda_jS_Wv_j. ]

Each eigenvector (v_j) defines a discriminant axis, and its eigenvalue measures the associated ratio of between-class to within-class scatter. Since the rank of (S_B) is at most (K-1), no more than (K-1) nonzero discriminant dimensions exist. The resulting subspace can consequently be much smaller than the original feature space even when the number of measurements is large.

Historical development

Fisher introduced the modern linear discriminant in 1936 while examining the taxonomic measurements later associated with the Iris flower data set. His construction emphasized separation through a linear combination of measurements rather than through isolated comparisons of individual variables. The formulation became a foundation for statistical classification because it connected estimation of group differences with the covariance structure of multivariate observations.

During the same period, Prasanta Chandra Mahalanobis developed the Mahalanobis distance, which measures displacement after accounting for covariance. Under the common-covariance normal model, assigning an observation by discriminant score is equivalent to comparing covariance-adjusted squared distances to class means, with an additional adjustment for class priors.

Subsequent work by C. R. Rao placed multiclass discrimination within a broader theory of canonical coordinates and multivariate inference. This development clarified the generalized eigenvalue structure of the problem and its relationship to multivariate analysis of variance. Later computational treatments recast the same algebra through matrix factorizations, allowing discriminant functions to be estimated without explicitly forming a numerically unstable matrix inverse.

Estimation and covariance structure

In applications, the population means and covariance matrix are replaced by estimates from labeled data. The pooled covariance estimator combines class-specific residual variation:

[ \widehat{\Sigma}

\frac{1}{n-K} \sum_{k=1}^{K} \sum_{i:Y_i=k} (x_i-\widehat{\mu}_k)(x_i-\widehat{\mu}_k)^\mathsf{T}. ]

The fitted discriminant scores are then obtained by substituting (\widehat{\mu}_k), (\widehat{\Sigma}), and estimated or externally specified priors into the population formula. Sampling error in these quantities affects both the orientation and location of the estimated decision boundaries.

The common-covariance assumption has a direct geometrical meaning: all classes have ellipsoidal density contours with the same orientation and shape. Their centers may differ, and their prior probabilities may differ, but a shared covariance causes pairwise log-posterior ratios to remain linear. When each class instead has its own covariance matrix (\Sigma_k), the discriminant score becomes

[ \delta_k(x)

-\frac{1}{2}\log|\Sigma_k| -\frac{1}{2}(x-\mu_k)^\mathsf{T}\Sigma_k^{-1}(x-\mu_k) +\log\pi_k. ]

Comparisons between these scores produce quadratic boundaries, giving quadratic discriminant analysis. The additional covariance parameters increase the model's capacity to represent class-specific dispersion while also increasing its sampling variability.

When the number of variables approaches or exceeds the number of observations, the sample within-class covariance matrix becomes unstable or singular. Regularized discriminant analysis, introduced by Jerome H. Friedman, replaces unrestricted covariance estimation with a structured compromise. One component shrinks class-specific covariance matrices toward a pooled estimate, while another shrinks the resulting matrix toward a scalar multiple of the identity. These modifications change the implied geometry from fully empirical covariance ellipsoids toward more constrained forms.

Relation to other statistical models

Linear discriminant analysis and logistic regression can produce decision boundaries of the same algebraic form, but they estimate different components of the joint distribution. Discriminant analysis models the distribution of predictors within each class and derives posterior probabilities through Bayes' theorem. Logistic regression models the conditional class probability directly, without specifying a probability distribution for the predictors.

Under the common-covariance normal model, the log odds between two classes are linear in (x), so the conditional probability has the logistic form. Differences arise when the normal model is inaccurate, when covariance estimates are unstable, or when the conditional model is correctly specified despite a more complicated predictor distribution. The distinction is therefore one of statistical modeling rather than merely one of boundary shape.

Discriminant analysis is also related to principal component analysis, although their optimization criteria differ. Principal component analysis identifies directions of large total variance without using class labels. Fisher-type discrimination uses the labels and compares between-class scatter with within-class scatter, so a direction containing little total variation can still be discriminative when its within-class variation is smaller still.

The generalized eigenvalue formulation further connects discriminant coordinates with canonical correlation. If class membership is represented through indicator variables, the nonzero canonical directions between the measurements and the indicator matrix span the same discriminant subspace under the standard weighting conventions. This equivalence links classification geometry to the broader theory of linear dependence between multivariate variable sets.

Interpretation and limitations

The coefficients of a discriminant function depend on measurement scale and on correlations among variables. A large coefficient does not independently establish that the corresponding variable has a large marginal class difference, because coefficients represent conditional contributions within the full covariance-adjusted combination. Structure coefficients, defined as correlations between original variables and discriminant scores, describe a different aspect of the fitted projection and need not rank variables in the same order.

Model performance is determined jointly by class overlap, parameter estimation, and the rule used to convert scores into decisions. Apparent accuracy calculated from the same observations used for fitting incorporates optimization to that sample. Estimates based on cross-validation or an independent test sample instead represent the behavior of the fitted procedure on observations excluded from parameter estimation.

Class imbalance enters the discriminant rule through prior probabilities and through the amount of information available for estimating each class distribution. Changing priors alters the intercept terms without changing the estimated covariance-adjusted directions. Changing the loss assigned to different errors alters the decision rule at the posterior stage and is mathematically distinct from changing the probability model.

See also