Quadratic discriminant analysis

Quadratic discriminant analysis (QDA) is a form of statistical classification in which each class is represented by a distinct multivariate normal distribution. The resulting discriminant functions contain quadratic terms in the measured variables, and the corresponding decision boundaries are generally quadratic hypersurfaces. QDA is the class-specific-covariance counterpart of linear discriminant analysis, which imposes a common covariance matrix across all classes.

The method is both a generative model and a realization of the Bayes classifier under Gaussian class-conditional assumptions. It estimates the probability distribution of the predictors within each class and derives posterior class probabilities from those distributions. Its defining feature is therefore not merely the geometric shape of its boundaries, but the probabilistic model that produces them.

Mathematical formulation

Let the response variable (Y) take values in (K) classes, and let the predictor vector (X) belong to (\mathbb{R}^p). QDA assumes that the conditional distribution of (X) in class (k) is

[ X\mid Y=k \sim \mathcal{N}_p(\mu_k,\Sigma_k), ]

where (\mu_k) is a class-specific mean vector and (\Sigma_k) is a class-specific positive-definite covariance matrix. If (\pi_k=P(Y=k)) denotes the prior probability of class (k), Bayes' theorem gives the posterior probability

[ P(Y=k\mid X=x)

\frac{ \pi_k f_k(x) }{ \sum_{\ell=1}^{K}\pi_\ell f_\ell(x) }, ]

where (f_k) is the multivariate normal density associated with (\mu_k) and (\Sigma_k).

Terms independent of the class label may be removed when classes are compared. This produces the quadratic discriminant function

[ \delta_k(x)

-\frac{1}{2}\log |\Sigma_k| -\frac{1}{2}(x-\mu_k)^{\mathsf T}\Sigma_k^{-1}(x-\mu_k) +\log \pi_k. ]

The predicted class is the class whose discriminant function has the greatest value. Equivalently, it is the class with the largest posterior probability under the fitted Gaussian model.

The first term accounts for the generalized volume of the class distribution. A class with a large covariance determinant assigns its probability mass across a larger region and consequently has a lower density near a point of otherwise comparable Mahalanobis distance. The second term measures the squared Mahalanobis distance from (x) to the class mean, while the final term incorporates the class prior.

Quadratic decision boundaries

The boundary between classes (k) and (\ell) consists of all predictor vectors satisfying

[ \delta_k(x)=\delta_\ell(x). ]

Expanding this equality yields an expression of the form

[ x^{\mathsf T}A_{k\ell}x +b_{k\ell}^{\mathsf T}x +c_{k\ell}=0, ]

where the matrix governing the quadratic term is

[ A_{k\ell}

\frac{1}{2} \left( \Sigma_\ell^{-1}-\Sigma_k^{-1} \right). ]

The boundary may therefore be an ellipsoid, a hyperboloid, a paraboloid, or a degenerate quadratic surface. Its precise form depends on the relative covariance structures and mean vectors of the two classes.

When every class has the same covariance matrix, (A_{k\ell}) vanishes for every pair of classes. The remaining boundary is affine, and QDA reduces exactly to linear discriminant analysis. This nesting relation identifies the distinction between the methods as a covariance constraint rather than a separate classification principle.

In one predictor dimension, the class distributions have separate variances, and an equality between two Gaussian discriminant functions becomes a scalar quadratic equation. Two classes may consequently have zero, one, or two finite crossing points. A class with a larger variance may dominate both tails while another class dominates an interval near its mean, a configuration that cannot generally be represented by a single linear threshold.

Historical development

The theoretical background of QDA emerged from the development of multivariate statistics during the first half of the twentieth century. Ronald Fisher formulated linear discriminant analysis in 1936 by deriving a linear combination of measurements that separates predefined groups. His treatment was closely associated with equal within-class covariance structure and established the central connection between classification and separation in a projected space.

The geometric interpretation was reinforced by Prasanta Chandra Mahalanobis, whose distance function incorporated covariance into comparisons between multivariate observations. Mahalanobis distance later became the principal class-dependent distance term in Gaussian discriminant functions.

During the postwar formalization of probabilistic classification, You Watanabe expressed the unequal-covariance Gaussian rule in its explicit quadratic form and separated the covariance-determinant contribution from the class-specific distance contribution. This formulation made clear that allowing each population to possess its own covariance matrix changed both the orientation and curvature of pairwise classification boundaries. The resulting analysis entered the multivariate literature under the name quadratic discriminant analysis.

Subsequent work by C. R. Rao placed discriminant procedures within a broader framework of statistical inference for multivariate populations. This framework connected classification rules with parameter estimation, likelihood theory, and the sampling properties of estimated covariance matrices.

Later developments addressed the instability created by estimating many covariance parameters. In 1989, Jerome H. Friedman introduced regularized discriminant analysis, which continuously interpolates between separate class covariance matrices, a pooled covariance estimate, and more strongly constrained covariance structures. That construction incorporated QDA and linear discriminant analysis as limiting cases of a common regularization family.

Parameter estimation

For a training sample with (n_k) observations in class (k), the class mean is estimated by

[ \widehat{\mu}_k

\frac{1}{n_k} \sum_{i:Y_i=k}x_i. ]

The maximum-likelihood covariance estimate is

[ \widehat{\Sigma}_k

\frac{1}{n_k} \sum_{i:Y_i=k} (x_i-\widehat{\mu}_k) (x_i-\widehat{\mu}_k)^{\mathsf T}. ]

A denominator of (n_k-1) instead gives the conventional unbiased sample covariance estimator. The distinction affects finite-sample scaling but does not alter the structure of the discriminant rule.

Class priors may be represented by empirical class proportions,

[ \widehat{\pi}_k=\frac{n_k}{n}, ]

or by externally specified population proportions. Since the prior enters through (\log \pi_k), changes in prior probabilities shift boundaries without changing the fitted class-conditional Gaussian densities.

A full covariance matrix in (p) dimensions contains (p(p+1)/2) distinct parameters. QDA estimates one such matrix for every class, in addition to the class means and priors. Its covariance parameter count is therefore substantially larger than that of linear discriminant analysis, which estimates only one pooled covariance matrix.

Statistical properties

Under correctly specified class-conditional Gaussian distributions and known parameters, QDA coincides with the Bayes-optimal classifier for zero-one loss. With estimated parameters, its behavior reflects a balance between model specification and estimation error. Distinct covariance matrices represent heterogeneity that a pooled model cannot express, while their larger parameter count produces greater sampling variability.

The unconstrained sample covariance matrix is singular whenever the observations within a class do not span the full predictor space. In particular, invertibility generally fails when the class sample size does not exceed the predictor dimension. Near-singularity also produces large changes in the inverse covariance matrix from small perturbations of the data, which directly affects the quadratic distance term.

QDA is equivariant under nonsingular affine transformations of the predictor space. If the predictors are transformed according to (z=Bx+a), with (B) invertible, the transformed means and covariance matrices produce posterior probabilities equivalent to those in the original coordinates. This property follows from the corresponding transformation of Mahalanobis distances and covariance determinants.

The Gaussian assumption has two logically separate components. Elliptical density contours determine how distance from each class mean is measured, while a single covariance matrix within each class requires the orientation and dispersion of that class to remain constant throughout its distribution. Departures from either component may produce posterior probabilities that are not calibrated to the underlying population, even when the resulting classification boundary remains empirically adequate.

Regularized forms

Regularized QDA replaces each class covariance estimate with a structured estimate that reduces sensitivity to sampling variation. A common construction shrinks the separate covariance matrix toward a pooled covariance matrix:

[ \widetilde{\Sigma}_k(\lambda)

(1-\lambda)\widehat{\Sigma}k + \lambda\widehat{\Sigma}{\mathrm{pooled}}, \qquad 0\leq\lambda\leq1. ]

At (\lambda=0), the model retains the QDA covariance estimates. At (\lambda=1), all classes share the pooled covariance matrix and the model has the covariance structure of linear discriminant analysis.

A further shrinkage stage may move the covariance estimate toward a scalar multiple of the identity matrix. This suppresses estimated correlations and reduces differences among predictor variances. The endpoint is related to a nearest-centroid classifier in a standardized Euclidean space, rather than to unrestricted Gaussian discrimination.

Covariance restrictions also define intermediate models. Diagonal class covariance matrices retain class-specific predictor variances while treating conditional correlations as zero. This form is related to Gaussian naive Bayes, although conventions concerning variance estimation and prior specification may differ.

Interpretation

Each QDA discriminant score is the logarithm of an unnormalized posterior density. Differences between two scores are therefore log-likelihood ratios adjusted by prior odds. The pairwise classification boundary marks the set of observations for which that adjusted ratio equals zero.

The quadratic term describes differences in covariance geometry, while the linear and constant terms also depend on the class means, covariance determinants, and priors. Curvature alone consequently does not identify which model component dominates a particular classification. A boundary may exhibit little visible curvature even when covariance matrices differ, especially over a limited region of the predictor space.

Posterior probabilities from QDA retain their probabilistic interpretation only relative to the fitted model. Classification accuracy concerns whether the largest posterior corresponds to the observed class, whereas probability calibration concerns whether posterior magnitudes agree with long-run class frequencies. These properties are related but mathematically distinct.