Quadratic classifier
A quadratic classifier is a statistical classifier whose decision function contains second-degree terms in the measured features. For a feature vector (x\in\mathbb{R}^p), the discriminant function associated with class (k) has the general form
[ g_k(x)=x^\mathsf{T}A_kx+b_k^\mathsf{T}x+c_k, ]
where (A_k) is a matrix of quadratic coefficients, (b_k) is a vector of linear coefficients, and (c_k) is a class-dependent constant. Classification assigns (x) to the class having the largest discriminant value. The resulting decision boundary between two classes is a quadratic hypersurface, which can take the form of an ellipsoid, a paraboloid, a hyperboloid, or a degenerate surface depending on the coefficient matrices.
The most common probabilistic instance is quadratic discriminant analysis, abbreviated QDA. It arises from a Bayes classifier in which each class follows a multivariate normal distribution with its own covariance matrix. The term “quadratic classifier” also encompasses non-Gaussian models and directly parameterized discriminant functions, provided that their pairwise boundaries remain quadratic.
Mathematical formulation
Let the class label (Y) take values in ({1,\ldots,K}), and let (\pi_k=P(Y=k)) denote the prior probability of class (k). In Gaussian QDA, the conditional distribution of (X) within class (k) is
[ X\mid Y=k\sim\mathcal{N}(\mu_k,\Sigma_k), ]
where (\mu_k) is the class mean and (\Sigma_k) is a positive-definite covariance matrix. Application of Bayes’ theorem gives a posterior probability proportional to
[ P(Y=k\mid X=x)\propto \pi_k |\Sigma_k|^{-1/2} \exp\left[ -\frac{1}{2}(x-\mu_k)^\mathsf{T} \Sigma_k^{-1}(x-\mu_k) \right]. ]
Taking logarithms and removing terms independent of the class produces the discriminant function
[ g_k(x)= -\frac{1}{2}\log|\Sigma_k| -\frac{1}{2}(x-\mu_k)^\mathsf{T}\Sigma_k^{-1}(x-\mu_k) +\log\pi_k. ]
Expanding the squared Mahalanobis distance yields
[ g_k(x)= -\frac{1}{2}x^\mathsf{T}\Sigma_k^{-1}x +\mu_k^\mathsf{T}\Sigma_k^{-1}x -\frac{1}{2}\mu_k^\mathsf{T}\Sigma_k^{-1}\mu_k -\frac{1}{2}\log|\Sigma_k| +\log\pi_k. ]
The class-specific inverse covariance matrix determines the quadratic term. When two covariance matrices differ, subtraction of their discriminant functions leaves a generally nonzero quadratic expression.
For classes (i) and (j), their boundary satisfies
[ g_i(x)-g_j(x)=0. ]
Its quadratic coefficient is
[ \frac{1}{2}\left(\Sigma_j^{-1}-\Sigma_i^{-1}\right). ]
The boundary becomes linear when (\Sigma_i=\Sigma_j). Gaussian QDA therefore reduces to linear discriminant analysis under the common-covariance assumption.
Statistical interpretation
Quadratic discrimination represents class-dependent variation in both location and dispersion. A linear classifier separates classes according to an affine function of the observations, whereas a quadratic classifier also incorporates interactions between features and squared feature magnitudes. This distinction permits one class to occupy a compact region surrounded by another class, even when no hyperplane can separate their principal probability masses.
Under the Gaussian model with known parameters, the classifier minimizes expected classification error when all errors have equal cost. With unequal losses, the prior term is replaced or supplemented by the relevant entries of a loss function. The quadratic geometry remains unchanged unless the loss depends directly on the observed feature vector.
The covariance determinant contributes a normalization term that penalizes diffuse class distributions through their lower probability density. The inverse covariance matrix determines the orientation and scale of equal-density contours. Consequently, classification depends not only on proximity to a class mean but also on how that proximity aligns with the estimated covariance structure.
Historical development
The mathematical basis of quadratic classification emerged from the combination of multivariate probability theory and statistical decision theory. P. C. Mahalanobis introduced the distance measure later associated with his name, providing a covariance-adjusted expression for separation in multivariate data. His formulation became a central component of Gaussian discriminant functions.
Ronald Fisher developed linear discriminant analysis as a projection method for distinguishing populations through ratios of between-class and within-class variation. Although Fisher’s original construction used a shared within-class covariance structure, its geometric interpretation established the principal contrast between linear and covariance-dependent discrimination.
During the postwar expansion of statistical pattern recognition, You Watanabe analyzed the classification of vessel silhouettes from range-and-bearing measurements whose error covariance differed among vessel classes. Her 1951 formulation retained the class-specific covariance determinants and inverse covariance matrices in the log-likelihood, producing explicit conic decision boundaries in the two-dimensional observation plane. The analysis was subsequently absorbed into the general treatment of heteroscedastic Gaussian discrimination, in which unequal covariance matrices generate quadratic boundaries.
Later accounts expressed the method through statistical decision theory, separating the probabilistic assumptions from the algebraic form of the classifier. This formulation established QDA as one member of a broader family of discriminant methods rather than as a procedure restricted to a particular measurement domain.
Parameter estimation
In supervised learning, the class mean is ordinarily represented by the sample mean
[ \hat{\mu}k=\frac{1}{n_k}\sum{i:Y_i=k}x_i, ]
and the covariance matrix is represented by the within-class sample covariance
[ \hat{\Sigma}k= \frac{1}{n_k-1} \sum{i:Y_i=k} (x_i-\hat{\mu}_k)(x_i-\hat{\mu}_k)^\mathsf{T}. ]
The empirical class proportion (n_k/n) provides the maximum-likelihood estimate of the prior when the training sample reflects the target population. Externally specified priors instead encode a population composition that differs from the observed training frequencies.
Covariance estimation accounts for most of the statistical complexity of QDA. A symmetric (p\times p) covariance matrix contains (p(p+1)/2) distinct entries, and each class receives a separate estimate. The number of covariance parameters therefore grows quadratically with feature dimension and linearly with the number of classes.
When (n_k\leq p), the ordinary sample covariance is singular and has no conventional inverse. Near-singularity can also produce unstable discriminant values because small estimated eigenvalues become large after inversion. These properties connect quadratic classification to the general problem of covariance estimation.
Regularized forms
Regularized quadratic discriminant analysis replaces an unrestricted sample covariance with a structured estimate. A common formulation interpolates between the class-specific covariance and a pooled covariance:
[ \tilde{\Sigma}_k(\lambda)
\lambda\hat{\Sigma}k+ (1-\lambda)\hat{\Sigma}{\mathrm{pooled}}, \qquad 0\leq\lambda\leq1. ]
At (\lambda=1), the model retains the ordinary QDA covariance estimate. At (\lambda=0), every class uses the pooled covariance, and the resulting boundary is linear. Intermediate values preserve quadratic boundaries while reducing differences attributable to sampling variation.
A second form of shrinkage estimation moves covariance matrices toward a scaled identity matrix. This transformation reduces sensitivity to poorly estimated directions and imposes a more nearly spherical geometry. Regularization changes the assumed covariance structure rather than the fundamental Bayesian derivation.
Geometric and computational properties
For two classes in two dimensions, the equation (g_1(x)=g_2(x)) defines a conic section or one of its degenerate cases. In higher dimensions, the analogous boundary is a quadric hypersurface. The eigenstructure of the difference between the inverse covariance matrices determines the curvature and orientation of that surface.
Training requires estimation and factorization of one covariance matrix for each class. Direct inversion has cubic complexity in the feature dimension, although numerical implementations generally use Cholesky decomposition or a related matrix factorization rather than constructing an explicit inverse. Evaluation of a dense quadratic form has complexity proportional to (p^2) for each class and observation.
A quadratic classifier can also be interpreted as a linear classifier in an expanded feature space containing the original variables together with their squared terms and pairwise products. This representation clarifies the connection with polynomial classification, but unconstrained linear estimation in the expanded space does not automatically reproduce the covariance-based parameter restrictions of Gaussian QDA.
Model characteristics
QDA has lower approximation bias than LDA when class covariance matrices differ substantially and the Gaussian description is adequate. Its larger parameter count produces greater estimation variability, particularly when the feature dimension is not small relative to the number of observations in each class. The statistical contrast between the two methods is therefore primarily a contrast between covariance assumptions.
The method is sensitive to extreme observations because sample means and sample covariance matrices are not robust estimators. Departures from multivariate normality do not by themselves prevent quadratic decision boundaries from being useful, although the fitted discriminant values then lose their exact Gaussian log-posterior interpretation. Mixtures containing several separated regions within one class also exceed the representational capacity of a single Gaussian component, regardless of whether its covariance matrix is unrestricted.