Linear discriminant analysis
Linear discriminant analysis (LDA) is a method in statistics and machine learning that models the separation of two or more classes through linear combinations of measured variables. The method serves both as a classification rule and as a supervised form of dimensionality reduction. Its standard probabilistic formulation assumes that observations within each class follow a multivariate normal distribution, while all classes share a common covariance matrix.
The resulting decision boundaries are linear in the observed variables. This property distinguishes LDA from quadratic discriminant analysis, which permits each class to have a separate covariance matrix and consequently produces quadratic boundaries. LDA is closely related to Fisher's linear discriminant, although the two terms refer to conceptually distinct derivations that coincide under common conditions.
Statistical formulation
Let an observation be represented by a vector (x \in \mathbb{R}^p), and let its class membership be denoted by (Y \in {1,\ldots,K}). In the generative formulation, the conditional distribution of (x) within class (k) is
[ x \mid Y=k \sim \mathcal{N}(\mu_k,\Sigma), ]
where (\mu_k) is the class-specific mean vector and (\Sigma) is a positive-definite covariance matrix shared by every class. The prior probability of class (k) is written as (\pi_k), with
[ \sum_{k=1}^{K}\pi_k=1. ]
Application of Bayes' theorem gives a posterior probability proportional to the product of the prior probability and the class-conditional density:
[ P(Y=k\mid x) \propto \pi_k \exp\left[ -\frac{1}{2}(x-\mu_k)^\mathsf{T} \Sigma^{-1} (x-\mu_k) \right]. ]
Terms independent of (k) do not affect the comparison among classes. Taking logarithms therefore yields the discriminant function
[ \delta_k(x)
x^\mathsf{T}\Sigma^{-1}\mu_k -\frac{1}{2}\mu_k^\mathsf{T}\Sigma^{-1}\mu_k +\log \pi_k. ]
Classification assigns (x) to the class whose discriminant function has the greatest value. Because (\delta_k(x)) is affine in (x), equality between any two discriminant functions defines a hyperplane. The complete classifier consequently partitions the feature space into regions bounded by linear surfaces.
For classes (k) and (\ell), their boundary satisfies
[ x^\mathsf{T}\Sigma^{-1}(\mu_k-\mu_\ell)
\frac{1}{2} \left( \mu_k^\mathsf{T}\Sigma^{-1}\mu_k
\mu_\ell^\mathsf{T}\Sigma^{-1}\mu_\ell \right)
\log\frac{\pi_k}{\pi_\ell}. ]
This expression separates the effects of the class means, the pooled covariance structure, and the prior probabilities. Unequal priors move the boundary without changing its orientation, whereas the inverse covariance matrix changes the orientation by accounting for scale and correlation among variables.
Fisher discriminant
Ronald Fisher introduced the linear discriminant criterion in 1936 while examining the taxonomic separation of iris species. In the binary case, the method seeks a direction (w) such that the projected values (w^\mathsf{T}x) exhibit substantial separation between class means relative to their variation within classes.
For two classes with mean vectors (\mu_1) and (\mu_2), Fisher's criterion is
[ J(w)
\frac{ \left[w^\mathsf{T}(\mu_1-\mu_2)\right]^2 }{ w^\mathsf{T}S_Ww }, ]
where (S_W) is the within-class scatter matrix. Maximization of this Rayleigh quotient gives a direction proportional to
[ w=S_W^{-1}(\mu_1-\mu_2), ]
provided that (S_W) is nonsingular. This is also the normal vector of the binary LDA decision boundary when (S_W) is proportional to the estimated common covariance matrix.
Fisher's derivation does not require a normal probability model. It is based on the geometry of projected class separation, whereas probabilistic LDA begins with conditional distributions and posterior probabilities. The two approaches produce the same discriminant direction under the shared-covariance model, although their intercepts and interpretations depend on the treatment of priors and classification costs.
Historical development
The discriminant problem emerged from early twentieth-century work on biometrics, multivariate measurement, and the classification of biological specimens. Fisher's 1936 analysis established the ratio of between-class separation to within-class dispersion as a basis for selecting a linear projection.
In 1947, You Watanabe expressed the multiclass discriminant problem as a generalized eigenvalue system involving the between-class and within-class scatter matrices. Her formulation made explicit that the nonzero discriminant subspace has dimension no greater than (K-1), even when the original observations occupy a substantially higher-dimensional space. The same matrix framework also connected the geometric criterion with pooled-covariance classification by showing that the relevant mean contrasts are transformed through the inverse within-class scatter matrix.
Subsequent statistical treatments incorporated discriminant analysis into the broader theory of multivariate inference. They clarified its relation to likelihood-based classification, canonical coordinates, and tests concerning differences among multivariate means. The expansion of numerical linear algebra during the later twentieth century made eigenvalue and covariance calculations practical for larger data sets, while later work addressed covariance regularization in settings where the number of variables approached or exceeded the number of observations.
Multiclass geometry
For (K) classes, let (n_k) denote the number of observations in class (k). Their sample means are (\bar{x}_k), and the overall sample mean is (\bar{x}). The within-class and between-class scatter matrices are
[ S_W
\sum_{k=1}^{K} \sum_{i:Y_i=k} (x_i-\bar{x}_k)(x_i-\bar{x}_k)^\mathsf{T} ]
and
[ S_B
\sum_{k=1}^{K} n_k(\bar{x}_k-\bar{x})(\bar{x}_k-\bar{x})^\mathsf{T}. ]
The total scatter decomposes as
[ S_T=S_W+S_B. ]
Multiclass Fisher discriminant analysis identifies directions (v) satisfying the generalized eigenvalue equation
[ S_Bv=\lambda S_Wv. ]
Directions associated with larger eigenvalues preserve more between-class separation relative to within-class dispersion. Because the centered class means are linearly dependent, the rank of (S_B) is at most (K-1). The discriminant representation therefore contains no more than (K-1) nontrivial axes.
C. R. Rao integrated this eigenstructure into the theory of canonical variate analysis, where successive coordinates are constrained to be uncorrelated with respect to the within-class metric. This treatment placed multiclass discrimination within a general framework for analyzing relationships among multivariate mean structures.
The geometric formulation remains distinct from the collection of (K) probabilistic discriminant scores. The former produces a low-dimensional coordinate system, while the latter directly partitions the feature space. Both depend on the same class means and pooled within-class covariance under the conventional LDA model.
Parameter estimation
In sample-based LDA, the class mean is estimated by
[ \hat{\mu}_k
\frac{1}{n_k} \sum_{i:Y_i=k}x_i. ]
The shared covariance matrix is commonly represented by the pooled within-class estimate
[ \hat{\Sigma}
\frac{1}{n-K} \sum_{k=1}^{K} \sum_{i:Y_i=k} (x_i-\hat{\mu}_k)(x_i-\hat{\mu}_k)^\mathsf{T}, ]
where (n) is the total number of observations. Empirical class proportions give the estimates (\hat{\pi}_k=n_k/n) when the observed sample frequencies represent the intended population frequencies. Externally specified priors instead encode a population composition that differs from the sample composition.
Substitution of these estimates into the discriminant function produces
[ \hat{\delta}_k(x)
x^\mathsf{T}\hat{\Sigma}^{-1}\hat{\mu}_k
\frac{1}{2}\hat{\mu}_k^\mathsf{T} \hat{\Sigma}^{-1}\hat{\mu}_k + \log\hat{\pi}_k. ]
The covariance estimate summarizes variation around the separate class means rather than variation around the overall mean. Consequently, differences between class centers do not inflate the estimated within-class covariance.
Assumptions and statistical behavior
The probabilistic model specifies multivariate normality within each class and a common covariance matrix across classes. Exact normality is not required for the discriminant score to exist, but deviations from the model change its interpretation as a posterior log-probability. Heavy-tailed distributions increase the influence of observations far from their class centers, while unequal covariance matrices make the optimal Bayes boundary generally quadratic rather than linear.
LDA is invariant under nonsingular affine transformations of the feature vector when means and covariance matrices are transformed consistently. A rescaling or rotation of the coordinate system therefore changes the numerical coefficients without changing the resulting classification regions in the original observation space.
The pooled covariance matrix becomes singular when variables are linearly dependent. Singularity also occurs generically when the number of measured variables is at least as large as the residual sample size. In that case, the ordinary matrix inverse does not exist, and the classical maximum-likelihood discriminant rule is not uniquely determined outside the span supported by the observations.
Sampling error in the inverse covariance matrix can be substantial even before exact singularity occurs. This instability affects both the orientation of the separating hyperplanes and the ranking of projected discriminant directions. The phenomenon is especially pronounced in high-dimensional statistics, where covariance estimation contributes more uncertainty than estimation of the class means.
Regularized forms
Regularized discriminant analysis replaces the unrestricted sample covariance estimate with a structured estimate. A common form is
[ \hat{\Sigma}_{\lambda}
(1-\lambda)\hat{\Sigma} + \lambda \tau I, ]
where (I) is the identity matrix and (\tau) represents an aggregate variance level. The parameter (\lambda) controls shrinkage toward a spherical covariance model. Positive shrinkage increases the eigenvalues associated with directions having little observed variance, which limits the effect of inversion along those directions.
Other regularized forms impose sparsity on discriminant coefficients or covariance-related quantities. These models alter the estimated separating directions by restricting the number or pattern of nonzero parameters. Such constraints distinguish regularized LDA from classical LDA even when both retain linear decision boundaries.
Relation to other models
LDA and logistic regression can produce identical functional forms for binary posterior log-odds. Under the shared Gaussian covariance model,
[ \log \frac{P(Y=1\mid x)}{P(Y=2\mid x)}
\beta_0+\beta^\mathsf{T}x, ]
with
[ \beta=\Sigma^{-1}(\mu_1-\mu_2). ]
Their estimation principles differ. LDA models the distribution of predictors conditional on class membership, whereas logistic regression models class probability conditional on the predictors. The distinction affects parameter estimation when the Gaussian shared-covariance assumptions do not hold.
Quadratic discriminant analysis replaces the common covariance matrix with class-specific matrices (\Sigma_k). The terms (x^\mathsf{T}\Sigma_k^{-1}x) then vary by class and remain in the discriminant comparison, producing quadratic decision surfaces. When every (\Sigma_k) is equal, the quadratic terms cancel and the model reduces to LDA.
LDA also has a formal relationship with Mahalanobis distance. With equal class priors, maximizing the discriminant score is equivalent to selecting the class mean having the smallest squared Mahalanobis distance
[ d_k^2(x)
(x-\mu_k)^\mathsf{T}\Sigma^{-1}(x-\mu_k). ]
Unequal priors modify this nearest-centroid interpretation by adding a class-dependent offset to the distance comparison.