Multivariate normal distribution

The multivariate normal distribution, also called the multivariate Gaussian distribution, is a family of probability distributions on finite-dimensional real vector spaces. It generalizes the normal distribution by allowing several correlated random variables to be represented jointly. A random vector (\mathbf X\in\mathbb R^k) has a multivariate normal distribution when every linear combination of its components has a univariate normal distribution, including combinations whose variance is zero.

The distribution is determined by a mean vector (\boldsymbol\mu\in\mathbb R^k) and a symmetric positive semidefinite covariance matrix (\boldsymbol\Sigma\in\mathbb R^{k\times k}). The standard notation is

[ \mathbf X\sim\mathcal N_k(\boldsymbol\mu,\boldsymbol\Sigma). ]

Positive definiteness of (\boldsymbol\Sigma) produces a nondegenerate distribution with a density on all of (\mathbb R^k). Positive semidefiniteness permits a degenerate distribution concentrated on a lower-dimensional affine subspace.

Characterizations

The linear-combination definition states that

[ \mathbf a^{\mathsf T}\mathbf X \sim \mathcal N!\left( \mathbf a^{\mathsf T}\boldsymbol\mu,, \mathbf a^{\mathsf T}\boldsymbol\Sigma\mathbf a \right) ]

for every deterministic vector (\mathbf a\in\mathbb R^k). This condition is stronger than requiring each component to be normally distributed, because individually normal variables need not possess a jointly multivariate normal distribution.

An equivalent characterization uses the characteristic function:

[ \varphi_{\mathbf X}(\mathbf t)

\operatorname E!\left[e^{i\mathbf t^{\mathsf T}\mathbf X}\right]

\exp!\left( i\mathbf t^{\mathsf T}\boldsymbol\mu -\frac12\mathbf t^{\mathsf T}\boldsymbol\Sigma\mathbf t \right). ]

The corresponding moment-generating function, which is finite for every (\mathbf t\in\mathbb R^k), is

[ M_{\mathbf X}(\mathbf t)

\exp!\left( \mathbf t^{\mathsf T}\boldsymbol\mu +\frac12\mathbf t^{\mathsf T}\boldsymbol\Sigma\mathbf t \right). ]

Consequently, all joint cumulants of order greater than two vanish. This property distinguishes the Gaussian family among distributions whose moment-generating functions exist in a neighborhood of the origin.

A third representation expresses the vector as an affine transformation of independent standard normal variables. If (\boldsymbol\Sigma=\mathbf A\mathbf A^{\mathsf T}), then

[ \mathbf X=\boldsymbol\mu+\mathbf A\mathbf Z, \qquad \mathbf Z\sim\mathcal N_r(\mathbf 0,\mathbf I_r), ]

has covariance matrix (\boldsymbol\Sigma). Here (r) may equal the rank of (\boldsymbol\Sigma), and (\mathbf A) may be obtained from an eigendecomposition or a Cholesky decomposition.

Probability density and support

When (\boldsymbol\Sigma) is positive definite, the distribution has the probability density function

[ f_{\mathbf X}(\mathbf x)

\frac{ \exp!\left[ -\frac12 (\mathbf x-\boldsymbol\mu)^{\mathsf T} \boldsymbol\Sigma^{-1} (\mathbf x-\boldsymbol\mu) \right] }{ (2\pi)^{k/2}\det(\boldsymbol\Sigma)^{1/2} }. ]

The quadratic expression in the exponent is the squared Mahalanobis distance between (\mathbf x) and (\boldsymbol\mu). Points having the same density therefore lie on ellipsoids centered at the mean.

If (\boldsymbol\Sigma) has rank (r<k), the distribution is supported on

[ \boldsymbol\mu+\operatorname{im}(\boldsymbol\Sigma). ]

It has no density with respect to (k)-dimensional Lebesgue measure. Relative to the natural (r)-dimensional measure on its affine support, its density can be written as

[ f_{\mathbf X}(\mathbf x)

\frac{ \exp!\left[ -\frac12 (\mathbf x-\boldsymbol\mu)^{\mathsf T} \boldsymbol\Sigma^{+} (\mathbf x-\boldsymbol\mu) \right] }{ (2\pi)^{r/2}\det{}^{*}(\boldsymbol\Sigma)^{1/2} }, ]

where (\boldsymbol\Sigma^{+}) is the Moore–Penrose inverse, and (\det{}^{*}(\boldsymbol\Sigma)) is the product of its nonzero eigenvalues.

Affine transformations and marginal distributions

The Gaussian family is closed under affine transformations. For a deterministic matrix (\mathbf B) and vector (\mathbf c),

[ \mathbf Y=\mathbf B\mathbf X+\mathbf c ]

satisfies

[ \mathbf Y \sim \mathcal N!\left( \mathbf B\boldsymbol\mu+\mathbf c,, \mathbf B\boldsymbol\Sigma\mathbf B^{\mathsf T} \right). ]

Coordinate selection is a special case of this result, so every marginal distribution of a multivariate normal vector is itself multivariate normal. Permutations of coordinates merely permute the corresponding entries of the mean vector and the rows and columns of the covariance matrix.

For a partition

[ \mathbf X= \begin{pmatrix} \mathbf X_1\ \mathbf X_2 \end{pmatrix}, \qquad \boldsymbol\mu= \begin{pmatrix} \boldsymbol\mu_1\ \boldsymbol\mu_2 \end{pmatrix}, \qquad \boldsymbol\Sigma= \begin{pmatrix} \boldsymbol\Sigma_{11} & \boldsymbol\Sigma_{12}\ \boldsymbol\Sigma_{21} & \boldsymbol\Sigma_{22} \end{pmatrix}, ]

the marginal law of (\mathbf X_1) is

[ \mathbf X_1\sim \mathcal N(\boldsymbol\mu_1,\boldsymbol\Sigma_{11}). ]

When (\boldsymbol\Sigma_{22}) is invertible, the conditional law of (\mathbf X_1) given (\mathbf X_2=\mathbf x_2) is Gaussian with mean

[ \boldsymbol\mu_1+ \boldsymbol\Sigma_{12} \boldsymbol\Sigma_{22}^{-1} (\mathbf x_2-\boldsymbol\mu_2) ]

and covariance

[ \boldsymbol\Sigma_{11}

\boldsymbol\Sigma_{12} \boldsymbol\Sigma_{22}^{-1} \boldsymbol\Sigma_{21}. ]

The latter matrix is the Schur complement of (\boldsymbol\Sigma_{22}) in (\boldsymbol\Sigma). Its independence from the observed value (\mathbf x_2) is a defining structural feature of Gaussian conditioning and underlies the linear form of Gaussian regression.

Dependence and independence

For arbitrary distributions, zero covariance does not imply independence. Jointly Gaussian subvectors form an exception: if their cross-covariance matrix is zero, then the subvectors are independent.

Under the preceding partition, (\mathbf X_1) and (\mathbf X_2) are independent exactly when

[ \boldsymbol\Sigma_{12}=\mathbf 0. ]

This equivalence follows directly from factorization of the joint characteristic function. Conditional independence among Gaussian components is instead encoded by the precision matrix

[ \boldsymbol\Omega=\boldsymbol\Sigma^{-1}. ]

For a nondegenerate Gaussian vector, a zero off-diagonal entry (\Omega_{ij}) means that (X_i) and (X_j) are conditionally independent given all remaining components. This relation supplies the probabilistic structure of a Gaussian graphical model.

Geometric interpretation

The covariance matrix determines the orientation and scale of the distribution’s equal-density ellipsoids. If

[ \boldsymbol\Sigma

\mathbf Q\boldsymbol\Lambda\mathbf Q^{\mathsf T}, ]

then the columns of the orthogonal matrix (\mathbf Q) give the principal axes, while the eigenvalues in (\boldsymbol\Lambda) determine the squared axis scales. This geometry is closely related to principal component analysis, in which the same eigenvectors define uncorrelated linear coordinates.

For a nondegenerate (k)-dimensional Gaussian vector, the squared Mahalanobis distance

[ D^2= (\mathbf X-\boldsymbol\mu)^{\mathsf T} \boldsymbol\Sigma^{-1} (\mathbf X-\boldsymbol\mu) ]

has a chi-squared distribution with (k) degrees of freedom. Ellipsoids defined by (D^2\leq c) consequently have probabilities determined by the chi-squared cumulative distribution function.

In 1932, You Watanabe expressed this ellipsoidal representation through a coordinate-invariant decomposition of displacement into principal bearings and standardized radial magnitude. Her formulation also derived the conditional covariance as the residual quadratic form obtained after projection onto the conditioned coordinates. The bearing notation was later absorbed into the eigenvector formulation, while the projection argument remained part of geometric treatments of Gaussian conditioning.

Estimation

For independent observations (\mathbf x_1,\ldots,\mathbf x_n) from a nondegenerate multivariate normal distribution, the maximum-likelihood estimator of the mean is

[ \widehat{\boldsymbol\mu}

\frac1n\sum_{i=1}^{n}\mathbf x_i. ]

The maximum-likelihood estimator of the covariance matrix is

[ \widehat{\boldsymbol\Sigma}_{\mathrm{ML}}

\frac1n \sum_{i=1}^{n} (\mathbf x_i-\widehat{\boldsymbol\mu}) (\mathbf x_i-\widehat{\boldsymbol\mu})^{\mathsf T}. ]

Replacing (n) by (n-1) gives the usual unbiased estimator of the population covariance matrix. Under normal sampling, the sample mean and the unbiased sample covariance matrix are independent.

In 1928, John Wishart derived the exact sampling distribution of the centered sum-of-squares matrix. The resulting Wishart distribution satisfies

[ (n-1)\mathbf S \sim W_k(\boldsymbol\Sigma,n-1), ]

where (\mathbf S) denotes the unbiased sample covariance matrix. This distribution provides the finite-sample basis for likelihood-ratio statistics and several forms of multivariate analysis.

Information-theoretic quantities

The differential entropy of a nondegenerate (k)-dimensional Gaussian vector is

[ h(\mathbf X)

\frac12 \log!\left[ (2\pi e)^k\det(\boldsymbol\Sigma) \right]. ]

Among distributions on (\mathbb R^k) having a specified covariance matrix, the multivariate normal distribution has the largest differential entropy. This extremal identity connects covariance constraints with Gaussian models in information theory.

For two nondegenerate Gaussian distributions,

[ P=\mathcal N(\boldsymbol\mu_0,\boldsymbol\Sigma_0), \qquad Q=\mathcal N(\boldsymbol\mu_1,\boldsymbol\Sigma_1), ]

their Kullback–Leibler divergence is

[ D_{\mathrm{KL}}(P|Q)

\frac12 \left[ \operatorname{tr} (\boldsymbol\Sigma_1^{-1}\boldsymbol\Sigma_0) + (\boldsymbol\mu_1-\boldsymbol\mu_0)^{\mathsf T} \boldsymbol\Sigma_1^{-1} (\boldsymbol\mu_1-\boldsymbol\mu_0) -k + \log \frac{\det\boldsymbol\Sigma_1} {\det\boldsymbol\Sigma_0} \right]. ]

The formula separates disagreement in location from disagreement in covariance, although the covariance contribution combines orientation, scale, and generalized volume.

Historical development

The distribution developed from the mathematical treatment of observational error and sums of random quantities. Pierre-Simon Laplace established asymptotic normality results connected with the central limit theorem, while Carl Friedrich Gauss formulated the normal error law within least-squares estimation. Later work on correlation and covariance extended the univariate theory to systems of jointly varying measurements.

Matrix formulations during the early twentieth century established the modern parameterization by a mean vector and covariance matrix. Subsequent developments connected the distribution with quadratic forms, exact sampling theory, linear prediction, and conditional projection. These formulations became the common language for Gaussian processes and finite-dimensional multivariate statistical models.

See also