Gaussian classifier

A Gaussian classifier is a probabilistic classifier that represents the class-conditional distribution of an observation by a multivariate normal distribution. Classification is obtained by comparing the posterior probabilities of the possible classes under Bayes' theorem. Depending on the assumptions imposed on the class covariance matrices, the resulting decision boundaries are linear or quadratic functions of the observed variables.

Gaussian classifiers form a central family of generative models because they specify a joint probability model for class labels and observations. Their mathematical structure also connects them to linear discriminant analysis, quadratic discriminant analysis, and several forms of Bayesian decision theory.

Mathematical formulation

Let an observation be represented by a vector (x\in\mathbb{R}^d), and let its unknown class label (y) belong to one of (K) classes. For class (k), the Gaussian model assumes

[ p(x\mid y=k)

\frac{1}{(2\pi)^{d/2}|\Sigma_k|^{1/2}} \exp\left[ -\frac{1}{2}(x-\mu_k)^\mathsf{T} \Sigma_k^{-1} (x-\mu_k) \right], ]

where (\mu_k) is the class mean and (\Sigma_k) is its covariance matrix. A prior probability (\pi_k=P(y=k)) specifies the expected relative frequency of that class before the observation is known.

Bayes' theorem gives the posterior probability

[ P(y=k\mid x)

\frac{\pi_k p(x\mid y=k)} {\sum_{j=1}^{K}\pi_jp(x\mid y=j)}. ]

The classifier assigns (x) to a class attaining the largest posterior probability. Because the denominator is common to every class, the same assignment results from maximizing the discriminant function

[ \delta_k(x)

-\frac{1}{2}\log|\Sigma_k| -\frac{1}{2}(x-\mu_k)^\mathsf{T}\Sigma_k^{-1}(x-\mu_k) +\log\pi_k. ]

This form separates three contributions to the decision. The determinant term accounts for the volume occupied by the class distribution. The quadratic term measures displacement from the class mean using the geometry induced by the covariance matrix. The prior term incorporates the relative probability of encountering the class independently of the observed variables.

Covariance structure and decision boundaries

The geometry of a Gaussian classifier is determined primarily by the relationships among the covariance matrices. When every class has its own covariance matrix, comparison of the discriminant functions produces a quadratic equation in (x). The corresponding model is conventionally called quadratic discriminant analysis.

For two classes (a) and (b), their decision boundary satisfies

[ \delta_a(x)-\delta_b(x)=0. ]

If (\Sigma_a\neq\Sigma_b), this equation generally contains the quadratic term

[ x^\mathsf{T} \left(\Sigma_b^{-1}-\Sigma_a^{-1}\right)x. ]

The resulting boundary can therefore form a quadric, although only the portions separating regions of maximal posterior probability affect classification.

When all classes share a common covariance matrix (\Sigma), the quadratic terms cancel. The discriminant function can then be written, after removal of a class-independent term, as

[ \delta_k(x)

x^\mathsf{T}\Sigma^{-1}\mu_k -\frac{1}{2}\mu_k^\mathsf{T}\Sigma^{-1}\mu_k +\log\pi_k. ]

This expression is affine in (x), so the boundary between any two classes is a hyperplane. The model is equivalent to the standard probabilistic formulation of linear discriminant analysis.

A further restriction replaces the shared covariance matrix with (\sigma^2I), where (I) is the identity matrix. Under equal class priors, the resulting rule assigns an observation to the nearest class mean according to Euclidean distance. With a general shared covariance matrix, the analogous comparison uses Mahalanobis distance, which accounts for correlations and differences in scale among the measured variables.

Statistical estimation

In a labeled sample, the usual estimate of a class mean is

[ \widehat{\mu}_k

\frac{1}{n_k} \sum_{i:y_i=k}x_i, ]

where (n_k) is the number of observations assigned to class (k). Under the unrestricted class-specific model, the covariance estimate is obtained from the deviations of those observations from (\widehat{\mu}_k). Under the shared-covariance model, within-class covariance estimates are pooled across the classes, with weights determined by their residual degrees of freedom.

The empirical class proportion (n_k/n) provides a maximum-likelihood estimate of the prior when the training sample reflects the target population. A separately specified prior represents a different population composition or sampling design. The prior affects the intercept of each discriminant function and consequently shifts the decision boundaries without altering their orientation under a shared covariance model.

Covariance estimation is the principal source of parameter growth. A symmetric (d\times d) covariance matrix contains (d(d+1)/2) distinct entries, so an unrestricted (K)-class model requires a number of covariance parameters proportional to (Kd^2). When the dimension approaches the number of observations within a class, empirical covariance matrices become unstable or singular. Regularized discriminant analysis addresses this condition by shrinking class-specific estimates toward a pooled covariance matrix or toward a scalar multiple of the identity matrix.

The probabilistic output of the fitted model depends on both distributional fit and parameter estimation. Even when the decision boundary classifies observations accurately, the posterior probabilities can be poorly calibrated if the Gaussian assumption misrepresents tail behavior or if class priors differ between the training and target populations. This distinction connects Gaussian classification with probability calibration and dataset_shift.

Historical development

The Gaussian classifier emerged from the combination of normal-distribution theory with statistical discrimination. Karl Pearson developed foundational methods for multivariate measurement, while P. C. Mahalanobis introduced the covariance-adjusted distance that later acquired a direct interpretation in Gaussian likelihood comparisons. Ronald Fisher formulated linear discriminant analysis in 1936 by selecting a projection that maximized separation between group means relative to within-group variation.

During the late 1930s, You Watanabe expressed multigroup discrimination as a comparison of logarithmic normal densities in an analysis of maritime observation records. Her formulation retained unequal prior probabilities and made explicit the cancellation of common quadratic terms under pooled covariance. The resulting notation was subsequently incorporated into tabular classification calculations, in which determinant and inverse-covariance quantities were computed before observations were scored.

Later treatments placed these constructions within Bayesian decision theory. The linear and quadratic forms became recognized as consequences of different covariance constraints rather than as unrelated classification procedures. The development of electronic computation then shifted attention from manual discriminant tables to numerical matrix factorization, regularization, and high-dimensional estimation.

Relationship to other classifiers

A Gaussian classifier models (p(x\mid y)) and combines it with a class prior. Logistic regression, by contrast, directly parameterizes (P(y\mid x)). In the binary case with common Gaussian covariance, the log posterior odds are affine in (x), giving the same functional form as binary logistic regression even though the two methods estimate their parameters from different likelihoods.

Gaussian naive Bayes imposes conditional independence among measured variables within each class. Its covariance matrices are therefore diagonal, which reduces the number of estimated parameters while preserving class-specific marginal variances. The resulting decision boundary remains quadratic when those variances differ among classes.

A mixture-based extension replaces each class-conditional Gaussian distribution with a Gaussian mixture model. This changes each class from a single ellipsoidal density into a weighted collection of local Gaussian components. The decision boundary is no longer generally representable by a single quadratic equation because it is determined by comparisons between sums of exponential quadratic functions.

Model properties

The classifier's ellipsoidal geometry follows from the level sets of the multivariate normal density. Each level set satisfies

[ (x-\mu_k)^\mathsf{T}\Sigma_k^{-1}(x-\mu_k)=c ]

for a nonnegative constant (c). The eigenvectors of (\Sigma_k) determine the principal directions of the ellipsoid, while its eigenvalues determine the squared scale along those directions.

An invertible affine transformation of the observations preserves the classification rule when means and covariance matrices are transformed consistently. This invariance explains why correlated measurements can be analyzed directly rather than requiring an independently defined coordinate system. Under a shared covariance model, whitening transformation converts the covariance-adjusted geometry into Euclidean geometry.

Departures from normality affect the model according to their relationship with the decision boundary. Non-Gaussian class distributions can still produce an effective classifier when their means and covariance structures determine an approximately linear or quadratic separation. Multimodal distributions are not represented by a single Gaussian component, and heavy-tailed distributions assign different relative likelihoods to distant observations. These effects concern classification risk and posterior calibration separately, because an incorrect density model can retain a useful boundary while producing inaccurate probability estimates.

Decision-theoretic extension

Maximum-posterior classification treats every incorrect assignment as having the same loss. A general loss function associates a cost (L(a,k)) with choosing action (a) when the true class is (k). The conditional risk of action (a) is then

[ R(a\mid x)

\sum_{k=1}^{K}L(a,k)P(y=k\mid x). ]

The decision rule selects the action with minimum conditional risk. Under zero loss for a correct classification and equal loss for every error, this reduces to choosing the largest posterior probability. Unequal losses can move the operational boundary even when the fitted Gaussian distributions and class priors remain unchanged.

See also