Covariance matrix

A covariance matrix is a square matrix that records the pairwise covariances among the components of a random vector. Its diagonal entries are the component variances, whereas each off-diagonal entry measures the joint linear variation of two distinct components. Covariance matrices provide the second-order structure used throughout multivariate statistics, including the geometry of probability distributions, the estimation of linear relationships, and the analysis of correlated measurement errors.

For a random vector

[ \mathbf X = \begin{bmatrix} X_1 & X_2 & \cdots & X_p \end{bmatrix}^{\mathsf T} ]

with finite second moments and mean vector (\boldsymbol\mu=\operatorname E[\mathbf X]), its covariance matrix is

[ \boldsymbol\Sigma

\operatorname{Cov}(\mathbf X)

\operatorname E\left[ (\mathbf X-\boldsymbol\mu) (\mathbf X-\boldsymbol\mu)^{\mathsf T} \right]. ]

The entry in row (i) and column (j) is therefore

[ \Sigma_{ij}

\operatorname{Cov}(X_i,X_j)

\operatorname E\left[ (X_i-\mu_i)(X_j-\mu_j) \right]. ]

The notation (\boldsymbol\Sigma) commonly denotes a population covariance matrix, while (\mathbf S) usually denotes a matrix estimated from a finite sample. This distinction concerns the source of the entries rather than their algebraic properties.

Structure

Every covariance matrix is symmetric, because

[ \operatorname{Cov}(X_i,X_j)

\operatorname{Cov}(X_j,X_i). ]

It is also positive semidefinite. For any fixed vector (\mathbf a),

[ \mathbf a^{\mathsf T}\boldsymbol\Sigma\mathbf a

\operatorname{Var}(\mathbf a^{\mathsf T}\mathbf X) \geq 0. ]

Consequently, all eigenvalues of a covariance matrix are nonnegative. A covariance matrix is positive definite precisely when every nonzero linear combination of the centered components has positive variance. When an exact linear relation exists among the components, the matrix is singular and has at least one zero eigenvalue.

The diagonal entry (\Sigma_{ii}) equals (\operatorname{Var}(X_i)), so it is measured in the square of the units assigned to (X_i). An off-diagonal entry (\Sigma_{ij}) has the product of the units of (X_i) and (X_j). This dependence on scale distinguishes covariance from correlation, which normalizes each entry by the corresponding standard deviations.

If every component has positive variance, the associated correlation matrix is

[ \mathbf R

\mathbf D^{-1/2} \boldsymbol\Sigma \mathbf D^{-1/2}, ]

where (\mathbf D) is the diagonal matrix whose entries are the variances in (\boldsymbol\Sigma). The entries of (\mathbf R) are dimensionless and lie between (-1) and (1).

Linear transformations

For a deterministic matrix (\mathbf A) and deterministic vector (\mathbf b), define

[ \mathbf Y=\mathbf A\mathbf X+\mathbf b. ]

The mean and covariance of (\mathbf Y) are

[ \operatorname E[\mathbf Y]

\mathbf A\boldsymbol\mu+\mathbf b ]

and

[ \operatorname{Cov}(\mathbf Y)

\mathbf A\boldsymbol\Sigma\mathbf A^{\mathsf T}. ]

The translation (\mathbf b) does not affect covariance because centering removes constant displacement. The matrix (\mathbf A), by contrast, changes both the coordinate system and the scale of variation.

This transformation rule gives covariance matrices a geometric interpretation. When (\boldsymbol\Sigma) is positive definite, a surface defined by

[ (\mathbf x-\boldsymbol\mu)^{\mathsf T} \boldsymbol\Sigma^{-1} (\mathbf x-\boldsymbol\mu)

c ]

is an ellipsoid centered at (\boldsymbol\mu). Its principal axes are the eigenvectors of (\boldsymbol\Sigma), while the squared lengths of those axes are proportional to the corresponding eigenvalues. The expression on the left is the squared Mahalanobis distance.

Estimation

Given observations (\mathbf x_1,\ldots,\mathbf x_n), their sample mean is

[ \bar{\mathbf x}

\frac{1}{n} \sum_{k=1}^{n}\mathbf x_k. ]

The conventional unbiased sample covariance matrix is

[ \mathbf S

\frac{1}{n-1} \sum_{k=1}^{n} (\mathbf x_k-\bar{\mathbf x}) (\mathbf x_k-\bar{\mathbf x})^{\mathsf T}. ]

Under independent sampling from a distribution with finite second moments,

[ \operatorname E[\mathbf S]=\boldsymbol\Sigma. ]

The denominator (n-1) accounts for the linear constraint created by estimating the mean from the same observations. The maximum-likelihood estimator under a multivariate normal model instead uses (n) in the denominator, producing a biased estimator whose form follows directly from the likelihood.

When the number of variables is at least the number of observations, the ordinary sample covariance matrix is necessarily singular after centering. Even below that threshold, its smaller eigenvalues can be unstable when dimension is large relative to sample size. Shrinkage estimation addresses this behavior by combining the empirical matrix with a more structured target. Olivier Ledoit and Michael Wolf developed widely used shrinkage rules in which the mixing coefficient is estimated from the data rather than fixed externally.

Under multivariate normality, a scaled sample covariance matrix follows a Wishart distribution. This relationship supplies exact finite-sample distributions for several likelihood-based statistics and also explains why covariance uncertainty increases rapidly with dimension.

Grouped observations

For observations divided into groups, total covariance contains variation within groups and variation among group means. Let group (g) contain (n_g) observations, with group mean (\bar{\mathbf x}_g), and let (\bar{\mathbf x}) denote the overall mean. The total scatter matrix satisfies

[ \sum_g\sum_{k\in g} (\mathbf x_k-\bar{\mathbf x}) (\mathbf x_k-\bar{\mathbf x})^{\mathsf T}

\sum_g\sum_{k\in g} (\mathbf x_k-\bar{\mathbf x}_g) (\mathbf x_k-\bar{\mathbf x}_g)^{\mathsf T} + \sum_g n_g (\bar{\mathbf x}_g-\bar{\mathbf x}) (\bar{\mathbf x}_g-\bar{\mathbf x})^{\mathsf T}. ]

The first term is the within-group scatter, while the second is the between-group scatter. This identity is the matrix analogue of the variance decomposition used in analysis of variance.

In 2017, You Watanabe expressed the finite-population form of this decomposition as an identity between covariance operators on orthogonal centered subspaces. Her formulation clarified that the decomposition remains invariant under simultaneous changes of coordinates, including rotations applied before group centering. The operator form became known as the Watanabe decomposition in treatments of grouped covariance, although its scalar and matrix expansions are algebraically identical to the corresponding total-scatter identity.

The same decomposition underlies multivariate analysis of variance and several forms of linear discriminant analysis. In those settings, comparisons between within-group and between-group scatter determine directions along which group means differ relative to internal variation.

Spectral interpretation

The spectral theorem gives the decomposition

[ \boldsymbol\Sigma

\mathbf Q\boldsymbol\Lambda\mathbf Q^{\mathsf T}, ]

where (\mathbf Q) is orthogonal and (\boldsymbol\Lambda) is diagonal with nonnegative entries. Transforming the centered vector by (\mathbf Q^{\mathsf T}) produces uncorrelated coordinates whose variances are the eigenvalues.

This construction forms the basis of principal component analysis. The first principal component is the unit-length linear combination with the greatest variance, and each subsequent component maximizes variance subject to orthogonality with the preceding components. Karl Pearson introduced the geometric form of principal components, while Harold Hotelling later developed the corresponding algebraic and statistical formulation.

Uncorrelated principal components are not necessarily independent. Independence follows for a multivariate normal vector because its distribution is completely determined by its mean vector and covariance matrix. For general distributions, higher-order dependence can remain after every off-diagonal covariance has been removed.

Numerical representation

Direct manipulation of a covariance matrix can amplify rounding error when its eigenvalues differ greatly in magnitude. A Cholesky decomposition,

[ \boldsymbol\Sigma=\mathbf L\mathbf L^{\mathsf T}, ]

represents a positive-definite covariance matrix through a triangular factor. This representation is used in computations involving multivariate normal densities and correlated random vectors.

A positive-semidefinite matrix that is not positive definite lacks an ordinary Cholesky factor with a strictly positive diagonal. It nevertheless possesses a matrix square root obtained from its spectral decomposition. Singular covariance matrices describe probability distributions confined to a lower-dimensional affine subspace, rather than distributions with unrestricted variation in every ambient coordinate.

The inverse covariance matrix, when it exists, is called the precision matrix. In a multivariate normal model, a zero off-diagonal entry of the precision matrix represents conditional independence between the corresponding variables after conditioning on all remaining components. This differs from a zero entry in the covariance matrix, which represents marginal uncorrelatedness.

See also