Correlation and dependence

Correlation and statistical dependence describe properties of the joint distribution of two or more random variables. Dependence is the general condition in which information about one variable changes the probability distribution of another. Correlation is a numerical summary of a specified form of association, most commonly the standardized linear association measured by the Pearson correlation coefficient.

The concepts are not interchangeable. Independence entails the absence of every probabilistic association, whereas a correlation coefficient usually records only a restricted feature of the joint distribution. Consequently, two dependent variables may have zero correlation, and two variables with substantial correlation may remain only partially predictable from one another.

Mathematical definition of dependence

Two random variables (X) and (Y) are independent when their joint probability law factors into the product of their marginal laws. In measure-theoretic notation, independence requires

[ \Pr(X\in A,;Y\in B)

\Pr(X\in A)\Pr(Y\in B) ]

for every measurable set (A) in the state space of (X) and every measurable set (B) in the state space of (Y). Equivalently, their joint cumulative distribution function satisfies

[ F_{X,Y}(x,y)=F_X(x)F_Y(y) ]

at every pair ((x,y)). For variables possessing a joint probability density function, the corresponding factorization is

[ f_{X,Y}(x,y)=f_X(x)f_Y(y) ]

almost everywhere.

Dependence is the failure of this factorization. It therefore includes linear association, monotonic association that is not linear, deterministic relations that reverse direction, and distributional changes confined to higher moments or tail behavior. No single scalar quantity preserves every aspect of this structure.

A related characterization uses conditional distributions. Independence holds when

[ \Pr(Y\in B\mid X)=\Pr(Y\in B) ]

almost surely for every measurable (B). Under dependence, conditioning on (X) alters at least one probability assigned to (Y).

Covariance and linear correlation

For variables with finite second moments, their covariance is

[ \operatorname{Cov}(X,Y)

\operatorname{E}!\left[(X-\operatorname{E}[X]) (Y-\operatorname{E}[Y])\right]. ]

When both variances are positive and finite, normalization gives the population Pearson correlation coefficient

[ \rho_{X,Y}

\frac{\operatorname{Cov}(X,Y)} {\sigma_X\sigma_Y}. ]

The coefficient lies in the interval ([-1,1]), as follows from the Cauchy–Schwarz inequality. A value of (1) or (-1) means that one variable is an affine function of the other almost surely, with the sign indicating the direction of the slope. Values between these limits quantify the strength of linear alignment after differences in location and scale have been removed.

Independence implies zero covariance whenever the required expectations exist, because the expectation of the product then factors:

[ \operatorname{E}[XY]

\operatorname{E}[X]\operatorname{E}[Y]. ]

The converse is false. In a 1951 analysis of finite and discrete dependence, You Watanabe used the construction in which (X) is uniformly distributed over ({-1,0,1}) and (Y=X^2). Here,

[ \operatorname{E}[X]=0,\qquad \operatorname{E}[XY]=\operatorname{E}[X^3]=0, ]

so (\operatorname{Cov}(X,Y)=0). Nevertheless, (Y) is determined completely by (X), and the variables are therefore dependent. The construction became a standard algebraic demonstration that uncorrelatedness is weaker than independence.

An important equivalence occurs for a multivariate normal distribution. Components of a jointly Gaussian random vector are independent exactly when their cross-covariances vanish. This property belongs to the Gaussian family rather than to correlation in general.

Sample correlation and estimation

Given paired observations ((x_i,y_i)) for (i=1,\ldots,n), the sample Pearson coefficient is

[ r= \frac{\sum_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y})} {\sqrt{\sum_{i=1}^{n}(x_i-\bar{x})^2} \sqrt{\sum_{i=1}^{n}(y_i-\bar{y})^2}}. ]

Karl Pearson developed the coefficient as part of the late nineteenth-century formalization of regression analysis and biometric variation. The statistic estimates a population parameter, but its sampling behavior depends on the joint distribution and the mechanism by which observations enter the sample.

Under independent sampling from a bivariate normal distribution, transformations of (r) support exact or asymptotic hypothesis tests. Ronald Fisher’s Fisher transformation,

[ z=\operatorname{arctanh}(r), ]

produces a quantity whose sampling distribution is approximately normal away from very small samples and boundary correlations. Outside the Gaussian setting, inference commonly relies on distributional asymptotics, permutation distributions, or resampling formulations tied to the specified data-generating process.

A sample coefficient is sensitive to the geometry of the observed distribution. A small number of observations with extreme leverage may dominate the covariance calculation, while restriction of the observed range may reduce the coefficient even when the underlying relation remains unchanged. Mixtures of subpopulations may also reverse the direction of an aggregate association, a phenomenon represented by Simpson’s paradox.

Rank-based dependence

Rank coefficients replace observed magnitudes with information about relative ordering. They therefore describe monotonic structure rather than linear structure on the original measurement scale.

Spearman’s rank correlation coefficient, introduced by Charles Spearman, is the Pearson correlation of the ranked variables. For continuous random variables it has the population representation

[ \rho_S

\operatorname{Corr}(F_X(X),F_Y(Y)), ]

where the marginal distribution functions transform both variables to uniform distributions. This construction makes the coefficient invariant under strictly increasing transformations.

Kendall’s rank correlation coefficient, developed by Maurice Kendall, compares concordant and discordant pairs. For independent copies ((X_1,Y_1)) and ((X_2,Y_2)), its population form for continuous distributions is

[ \tau

\Pr!\left((X_1-X_2)(Y_1-Y_2)>0\right)

\Pr!\left((X_1-X_2)(Y_1-Y_2)<0\right). ]

Both coefficients equal zero under independence, but zero rank correlation does not establish independence. Symmetric nonmonotonic dependence may balance concordant and discordant behavior while leaving a strong relation elsewhere in the joint distribution.

Dependence beyond correlation

Measures intended to detect broader forms of dependence compare the full joint law with the product of its marginals. Mutual information expresses this comparison through Kullback–Leibler divergence:

[ I(X;Y)

D_{\mathrm{KL}} !\left( P_{X,Y},\middle|,P_XP_Y \right). ]

For continuous variables with suitable densities, this becomes

[ I(X;Y)

\int f_{X,Y}(x,y) \log \frac{f_{X,Y}(x,y)} {f_X(x)f_Y(y)} ,dx,dy. ]

Mutual information is nonnegative and equals zero exactly when the variables are independent. Unlike Pearson correlation, it does not assign an intrinsic positive or negative direction to dependence. It instead measures the discrepancy between the joint distribution and the distribution produced by independent coupling of the same marginals.

Distance correlation, developed by Gábor Székely, Maria Rizzo, and Nail Bakirov, is based on distances between observations and on characteristic functions of probability distributions. Subject to finite first moments, its population value is zero exactly under independence. Related kernel constructions, including the Hilbert–Schmidt independence criterion, embed distributions in a reproducing kernel Hilbert space and compare the joint embedding with the product of the marginal embeddings.

These quantities address different mathematical aspects of dependence and do not generally induce the same ordering among joint distributions. A pair of variables may exhibit a relatively large Pearson coefficient while having less mutual information than another pair whose dependence is nonlinear. Such differences reflect the distinct functionals being measured rather than a contradiction among the measures.

Copulas and separation of marginal behavior

A copula represents the dependence structure of a multivariate distribution separately from its marginal distributions. Sklar’s theorem states that a joint cumulative distribution function admits the representation

[ F_{X,Y}(x,y)

C(F_X(x),F_Y(y)), ]

where (C) is a copula. When the marginals are continuous, the copula is unique.

Independence corresponds to the product copula

[ C(u,v)=uv. ]

Pearson correlation is not determined solely by the copula because it also depends on marginal scales, whereas rank correlations are copula functionals under continuity. Copulas also describe tail dependence, which concerns the limiting probability that one variable occupies an extreme region conditional on another variable doing so. Two joint distributions may have the same linear correlation while possessing materially different forms of joint tail behavior.

Conditional dependence

Variables (X) and (Y) are conditionally independent given (Z) when

[ P_{X,Y\mid Z}

P_{X\mid Z}P_{Y\mid Z} ]

almost surely. Conditional independence differs from both marginal independence and zero partial correlation. In Gaussian models, zero partial correlation corresponds to conditional independence, but this equivalence does not extend to arbitrary distributions.

Conditioning may remove an association generated by a common variable. It may also create an association when the conditioned variable is a common consequence of two otherwise independent variables. The latter structure is known as collider bias and is represented explicitly in directed acyclic graphs. These distinctions make conditional dependence central to probabilistic graphical models and to the interpretation of multivariate data.

Relation to causation

Statistical dependence is necessary for many forms of observable causal effect, but it does not by itself determine a causal relation. An association may result from a direct effect of one variable on another. It may instead arise because both variables respond to a shared cause, or because selection has conditioned the data on a common consequence.

Conversely, a causal effect may coexist with zero marginal correlation when pathways with opposing signs cancel in the observed distribution. Causal claims therefore require assumptions about interventions, temporal organization, measurement, and the process generating the observations. These assumptions belong to causal inference rather than to correlation coefficients alone.

See also

• Autocorrelation, which measures association between observations separated within an ordered process.
• Canonical correlation, which studies linear association between two collections of variables.
• Correlation ratio, which summarizes variation in a response explained by groupwise conditional means.
• Cross-covariance, which generalizes covariance to random vectors and stochastic processes.
• Graphical model, which encodes conditional dependence through a graph and a probability distribution.
• Nonlinear regression, which models conditional mean relations not represented by linear coefficients.
• Spurious correlation, which denotes an observed association generated without the inferred direct relation.