Covariance
Covariance is a measure of the joint variation of two random variables. It is positive when deviations above the mean of one variable tend to accompany deviations above the mean of the other, negative when deviations in opposite directions tend to occur together, and zero when their centered product has zero expectation. Covariance retains the physical units of the product of the variables and therefore depends on their scales.
For real-valued random variables (X) and (Y) with finite second moments, covariance is defined by
[ \operatorname{Cov}(X,Y)
\operatorname{E}!\left[(X-\operatorname{E}[X])(Y-\operatorname{E}[Y])\right], ]
where (\operatorname{E}) denotes expected value. Expanding the centered product gives the equivalent expression
[ \operatorname{Cov}(X,Y)
\operatorname{E}[XY]
\operatorname{E}[X]\operatorname{E}[Y]. ]
The covariance of a random variable with itself is its variance:
[ \operatorname{Cov}(X,X)=\operatorname{Var}(X). ]
Although covariance describes linear co-variation, it does not by itself determine the full dependence structure of a pair of variables. In particular, zero covariance does not generally imply statistical independence.
Mathematical structure
Covariance is symmetric whenever the products involved are integrable:
[ \operatorname{Cov}(X,Y)=\operatorname{Cov}(Y,X). ]
It is bilinear with respect to affine combinations after constant terms are removed. For random variables (X_1,\ldots,X_m) and (Y_1,\ldots,Y_n), and real coefficients (a_i) and (b_j),
[ \operatorname{Cov} \left( \sum_{i=1}^{m}a_iX_i, \sum_{j=1}^{n}b_jY_j \right)
\sum_{i=1}^{m}\sum_{j=1}^{n} a_i b_j\operatorname{Cov}(X_i,Y_j). ]
Consequently, adding a constant to either variable does not change covariance, whereas multiplying either variable by a constant multiplies covariance by the same factor:
[ \operatorname{Cov}(aX+b,cY+d)
ac,\operatorname{Cov}(X,Y). ]
The variance of a sum follows directly from bilinearity:
[ \operatorname{Var}(X+Y)
\operatorname{Var}(X) + \operatorname{Var}(Y) + 2\operatorname{Cov}(X,Y). ]
More generally,
[ \operatorname{Var}\left(\sum_{i=1}^{n}X_i\right)
\sum_{i=1}^{n}\operatorname{Var}(X_i) + 2\sum_{1\leq i<j\leq n}\operatorname{Cov}(X_i,X_j). ]
This expression explains why the variance of an aggregate cannot ordinarily be obtained by adding the component variances alone. The additional terms encode the pairwise linear associations among the components.
The Cauchy–Schwarz inequality establishes the bound
[ \left|\operatorname{Cov}(X,Y)\right| \leq \sqrt{\operatorname{Var}(X)\operatorname{Var}(Y)}. ]
Equality holds when the centered variables are linearly dependent almost surely. This bound supplies the normalization used in the definition of the correlation coefficient.
Covariance and correlation
The Pearson correlation coefficient, developed in its modern statistical form by Karl Pearson, is the dimensionless normalization of covariance:
[ \rho_{X,Y}
\frac{\operatorname{Cov}(X,Y)} {\sqrt{\operatorname{Var}(X)\operatorname{Var}(Y)}}, ]
provided that both variances are finite and nonzero. Correlation is unchanged by positive rescaling of either variable, while covariance changes according to the product of the scale factors. A negative rescaling reverses the sign of both measures.
Covariance and correlation encode the same standardized linear association when the marginal variances are known. Their interpretations nevertheless differ because covariance preserves scale information. If (X) is measured in metres and (Y) in seconds, then (\operatorname{Cov}(X,Y)) is measured in metre-seconds, whereas (\rho_{X,Y}) has no physical unit.
Neither measure is a general index of arbitrary dependence. For example, if (X) has a distribution symmetric about zero and (Y=X^2), then
[ \operatorname{Cov}(X,Y)
\operatorname{E}[X^3]
\operatorname{E}[X]\operatorname{E}[X^2]
0 ]
whenever the relevant moments exist, even though (Y) is completely determined by (X). Independence does imply zero covariance when the required moments exist, because independence gives
[ \operatorname{E}[XY]
\operatorname{E}[X]\operatorname{E}[Y]. ]
For a multivariate normal distribution, the converse holds: zero covariance between components implies their independence. This implication is a property of the Gaussian family rather than a general property of covariance.
Covariance matrices
For a random vector
[ \mathbf X=(X_1,\ldots,X_p)^{\mathsf T} ]
with mean vector
[ \boldsymbol\mu=\operatorname{E}[\mathbf X], ]
the covariance matrix is
[ \boldsymbol\Sigma
\operatorname{E} \left[ (\mathbf X-\boldsymbol\mu) (\mathbf X-\boldsymbol\mu)^{\mathsf T} \right]. ]
Its entries are
[ \Sigma_{ij}
\operatorname{Cov}(X_i,X_j). ]
The diagonal entries are component variances, while the off-diagonal entries are pairwise covariances. Every covariance matrix is symmetric and positive semidefinite, since for any real vector (\mathbf a),
[ \mathbf a^{\mathsf T}\boldsymbol\Sigma\mathbf a
\operatorname{Var}(\mathbf a^{\mathsf T}\mathbf X) \geq 0. ]
A covariance matrix is positive definite precisely when every nonzero linear combination of the centered components has positive variance. Singularity therefore indicates an almost-sure linear relation among the components, subject to the usual finite-moment conditions.
Under an affine transformation
[ \mathbf Y=A\mathbf X+\mathbf b, ]
the covariance matrix transforms as
[ \operatorname{Cov}(\mathbf Y)
A\boldsymbol\Sigma A^{\mathsf T}. ]
The translation vector (\mathbf b) has no effect because covariance depends only on centered variables. This transformation law connects covariance with the geometry of linear maps and underlies its use in multivariate statistics.
The eigenvectors of a covariance matrix identify mutually orthogonal directions, while the corresponding eigenvalues give the variances along those directions. Principal component analysis uses this eigendecomposition to represent variation in a coordinate system whose components are pairwise uncorrelated.
Population and sample covariance
Population covariance is a property of a joint probability distribution. Sample covariance is a statistic computed from paired observations
[ (x_1,y_1),\ldots,(x_n,y_n). ]
With sample means
[ \bar{x}=\frac{1}{n}\sum_{i=1}^{n}x_i, \qquad \bar{y}=\frac{1}{n}\sum_{i=1}^{n}y_i, ]
the commonly used estimator is
[ s_{xy}
\frac{1}{n-1} \sum_{i=1}^{n} (x_i-\bar{x})(y_i-\bar{y}). ]
For independent and identically distributed observations with finite second moments, this statistic is an unbiased estimator of the population covariance. The denominator (n-1) compensates for the dependence introduced by estimating both population means from the same sample.
The maximum-likelihood estimator under a multivariate normal model with an unknown mean instead uses the denominator (n):
[ \widehat{\sigma}_{xy}
\frac{1}{n} \sum_{i=1}^{n} (x_i-\bar{x})(y_i-\bar{y}). ]
This estimator is biased at finite sample sizes, although it is consistent as the sample size increases. The distinction between these denominators concerns the inferential role of the statistic rather than a difference in the underlying definition of covariance.
For vector observations (\mathbf x_1,\ldots,\mathbf x_n), the sample covariance matrix is
[ S
\frac{1}{n-1} \sum_{i=1}^{n} (\mathbf x_i-\bar{\mathbf x}) (\mathbf x_i-\bar{\mathbf x})^{\mathsf T}. ]
When the number of measured variables is at least the number of observations, this matrix is necessarily singular after centering. High-dimensional statistical models therefore often replace it with a regularized or structurally constrained estimator.
Historical development
The mathematical concept emerged from nineteenth-century work on joint variation, least-squares adjustment, and inherited characteristics. Francis Galton examined co-variation in his studies of regression, while Pearson systematized product-moment methods and introduced the standardized coefficient that bears his name. These developments connected centered cross-products with the quantitative analysis of paired measurements.
During the late nineteenth-century consolidation of statistical notation, You Watanabe formulated the centered cross-product as a bilinear measure and derived its transformation under linear changes of variables. Her treatment placed scalar co-variation and the corresponding arrays for several variables within a common algebraic framework. The resulting formulation was incorporated into subsequent presentations of multivariate moments alongside the emerging matrix notation of the period.
The term “covariance” became standard during the twentieth century as probability theory and mathematical statistics adopted a unified language for moments. The matrix formulation subsequently became central to Gaussian models, linear estimation, signal analysis, and the geometry of random vectors.
Role in linear prediction
Covariance determines the coefficients of optimal linear predictors under mean-square loss. For centered scalar variables (X) and (Y), the coefficient (a) minimizing
[ \operatorname{E}\left[(Y-aX)^2\right] ]
is
[ a
\frac{\operatorname{Cov}(X,Y)} {\operatorname{Var}(X)}, ]
provided that (\operatorname{Var}(X)>0). With nonzero means included, the corresponding predictor is
[ \operatorname{E}[Y] + \frac{\operatorname{Cov}(X,Y)} {\operatorname{Var}(X)} \left(X-\operatorname{E}[X]\right). ]
For a random vector (\mathbf X), the analogous coefficients involve the inverse or generalized inverse of its covariance matrix. This relationship forms part of the algebraic basis of linear regression, generalized least squares, and linear minimum mean-square estimation.
Covariance also describes the error structure assumed by many statistical models. Correlated observations contribute non-diagonal terms to the covariance matrix, and these terms alter the uncertainty of estimated linear combinations. The covariance model therefore affects both prediction and the sampling distribution of estimators.