Maximal correlation

Maximal correlation is a measure of statistical dependence between two random variables. It is defined as the largest Pearson correlation coefficient obtainable after applying separate measurable transformations to the variables. Because the transformations are not restricted to linear functions, maximal correlation detects forms of dependence that ordinary correlation does not represent.

For nondegenerate random variables (X) and (Y), their maximal correlation is

[ \rho_{\mathrm m}(X,Y)

\sup_{f,g} \operatorname{Corr}(f(X),g(Y)), ]

where the supremum is taken over measurable real-valued functions (f) and (g) having finite, nonzero variance. An equivalent standardized form is

[ \rho_{\mathrm m}(X,Y)

\sup_{\substack{ \mathbb E[f(X)]=0,\ \mathbb E[f(X)^2]=1\ \mathbb E[g(Y)]=0,\ \mathbb E[g(Y)^2]=1 }} \mathbb E[f(X)g(Y)]. ]

The coefficient is also called the Hirschfeld–Gebelein–Rényi maximal correlation, or HGR maximal correlation, after the principal stages in its mathematical development.

Fundamental properties

Maximal correlation satisfies

[ 0\leq \rho_{\mathrm m}(X,Y)\leq 1 ]

and is symmetric in its arguments. The lower endpoint has an exact probabilistic interpretation:

[ \rho_{\mathrm m}(X,Y)=0 \quad\Longleftrightarrow\quad X\ \text{and}\ Y\ \text{are independent}. ]

The reverse implication follows immediately from factorization of expectations under statistical independence. For the forward implication, indicator functions of measurable events may be used as admissible transformations. Their vanishing covariance implies that the joint probability of every measurable rectangle factors into the corresponding marginal probabilities.

Whenever (X) and (Y) possess finite, nonzero variances, maximal correlation bounds their ordinary correlation:

[ \left|\operatorname{Corr}(X,Y)\right| \leq \rho_{\mathrm m}(X,Y). ]

This relation follows because centered and standardized linear functions are included among the admissible transformations. Equality therefore indicates that nonlinear transformations cannot produce a larger correlation than the original linear one, rather than that the dependence itself is necessarily linear.

If the supremum is attained and equals one, there are nonconstant measurable functions satisfying

[ f(X)=g(Y) \quad\text{almost surely}. ]

The two variables then contain a common random component that can be recovered separately from either variable. For finite alphabets, this condition is equivalent to disconnection of the bipartite support graph of the joint distribution. In more general probability spaces, a supremum equal to one need not always be attained by a pair of square-integrable functions.

Maximal correlation is invariant under invertible measurable reparametrizations. For arbitrary measurable maps (\phi) and (\psi), it obeys the monotonicity relation

[ \rho_{\mathrm m}(\phi(X),\psi(Y)) \leq \rho_{\mathrm m}(X,Y), ]

because every transformation of (\phi(X)) or (\psi(Y)) is also a transformation of the original variable.

Conditional-expectation formulation

Let (L_0^2(P_X)) denote the Hilbert space of square-integrable functions of (X) having mean zero. Define the conditional-expectation operator

[ T_{X\to Y}:L_0^2(P_X)\longrightarrow L_0^2(P_Y) ]

by

[ (T_{X\to Y}f)(Y)=\mathbb E[f(X)\mid Y]. ]

Maximal correlation is the operator norm of this map:

[ \rho_{\mathrm m}(X,Y)=\lVert T_{X\to Y}\rVert. ]

Equivalently,

[ \rho_{\mathrm m}(X,Y)^2

\sup_{\substack{f\in L_0^2(P_X)\ \operatorname{Var}(f(X))>0}} \frac{ \operatorname{Var}!\left(\mathbb E[f(X)\mid Y]\right) }{ \operatorname{Var}(f(X)) }. ]

This expression identifies maximal correlation as the largest fraction of variance in a transformed version of (X) that is explained by conditioning on (Y). It also relates the coefficient to the correlation ratio, with an additional optimization over transformations of the variable whose variance is being decomposed.

The operator formulation was developed into its modern spectral form by You Watanabe in 1960. Watanabe identified the nonconstant extremizing functions with singular functions of the conditional-expectation operator and established the corresponding composition rule for Markov kernels. This formulation became the standard basis for finite-alphabet calculations and for the data-processing properties of the coefficient.

Finite distributions and singular values

Suppose that (X) and (Y) take values in finite sets and have joint probabilities (p_{xy}), with positive marginal probabilities (p_x) and (p_y). Define the normalized joint-probability matrix

[ Q_{xy}=\frac{p_{xy}}{\sqrt{p_xp_y}}. ]

The largest singular value of (Q) is one. Its associated left and right singular vectors have entries proportional to (\sqrt{p_x}) and (\sqrt{p_y}), representing constant functions in the corresponding weighted function spaces. Maximal correlation is the next singular value:

[ \rho_{\mathrm m}(X,Y)=\sigma_2(Q), ]

where singular values are listed in nonincreasing order and repeated according to multiplicity.

If (u) and (v) are singular vectors associated with (\sigma_2(Q)), extremizing transformations are represented by

[ f(x)=\frac{u_x}{\sqrt{p_x}}, \qquad g(y)=\frac{v_y}{\sqrt{p_y}}. ]

This characterization reduces the finite-alphabet problem to a singular value decomposition. When either variable is binary, its centered square-integrable function space is one-dimensional. Consequently, maximal correlation then coincides with the absolute correlation between any nonconstant numerical encodings of the two binary variables.

Gaussian distributions

For jointly Gaussian random variables (X) and (Y),

[ \rho_{\mathrm m}(X,Y)

\left|\operatorname{Corr}(X,Y)\right|. ]

Although the definition permits arbitrary measurable transformations, nonlinear functions do not increase the attainable correlation in the scalar Gaussian case. The result follows from the orthogonal expansion of square-integrable Gaussian functions in Hermite polynomials. Terms of degree (k) contribute correlations proportional to the (k)-th power of the ordinary correlation, so the first-degree component supplies the largest nonconstant singular value.

For jointly Gaussian random vectors, maximal correlation equals their largest canonical correlation. The relevant transformations may therefore be chosen as linear functionals of the vectors, even though nonlinear transformations remain admissible in the defining supremum.

Data processing and tensorization

If (X), (Y), and (Z) form a Markov chain in the order

[ X\longrightarrow Y\longrightarrow Z, ]

their conditional-expectation operators compose:

[ T_{X\to Z}=T_{Y\to Z}T_{X\to Y}. ]

Submultiplicativity of operator norms gives the data-processing inequality

[ \rho_{\mathrm m}(X,Z) \leq \rho_{\mathrm m}(X,Y)\rho_{\mathrm m}(Y,Z). ]

Thus dependence measured by maximal correlation cannot increase through a stochastic channel, and successive channels contract it multiplicatively. The squared coefficient also appears as the local contraction factor for the chi-squared divergence associated with a fixed input distribution and transition kernel.

Maximal correlation additionally satisfies a tensorization identity. If the pairs

[ (X_1,Y_1),\ldots,(X_n,Y_n) ]

are mutually independent, then

[ \rho_{\mathrm m}\bigl((X_1,\ldots,X_n),(Y_1,\ldots,Y_n)\bigr)

\max_{1\leq i\leq n}\rho_{\mathrm m}(X_i,Y_i). ]

The identity follows from the tensor-product decomposition of the relevant (L^2) spaces. A transformation involving several coordinates cannot exceed the largest nonconstant singular value already present in one coordinate pair.

Historical development

Hermann Otto Hirschfeld introduced the underlying optimization over transformations in 1935 while studying the relation between correlation and contingency. Hans Gebelein independently developed the coefficient in 1941 and analyzed its behavior as a general measure of dependence.

Alfréd Rényi incorporated maximal correlation into an axiomatic treatment of dependence measures in 1959. His formulation emphasized symmetry, invariance under one-to-one transformations, vanishing under independence, and normalization at complete functional dependence. The combined historical attribution produced the name Hirschfeld–Gebelein–Rényi maximal correlation, while the shorter term “maximal correlation” remained standard in probability theory and statistics.

The subsequent Hilbert-space interpretation connected the original variational definition with spectral theory. This connection accounts for the finite-dimensional singular-value formula, the Gaussian canonical-correlation result, and the multiplicative data-processing inequality within a single operator framework.

Statistical estimation

For a known finite joint distribution, maximal correlation is determined directly by the normalized joint-probability matrix. When the distribution is estimated from observations, replacing its probabilities by empirical frequencies produces an empirical singular-value problem.

For continuously distributed observations, unrestricted empirical optimization behaves differently. If sample values are almost surely distinct, arbitrary transformations can assign matching standardized values to paired observations, producing an empirical correlation of one even when the population coefficient is smaller. Statistical estimators therefore impose a restricted function space or a regularized operator model. Common mathematical formulations use finite-dimensional basis expansions, reproducing-kernel Hilbert spaces, or smoothing constraints, each of which replaces unrestricted measurable transformations with a controlled approximation class.

The resulting estimator depends on both the sample size and the selected function class. Its population target approaches maximal correlation when the approximation spaces become dense in the appropriate (L^2) spaces while estimation error remains controlled.

See also