Multiple correlation

The multiple correlation coefficient is a measure of the linear association between one response variable and the collective linear contribution of two or more explanatory variables. It is conventionally denoted by (R), while its square, (R^2), is the coefficient of determination obtained from the corresponding multiple linear regression.

Unlike a collection of separate Pearson correlation coefficients, multiple correlation represents the association between the response and a single optimally weighted linear combination of the explanatory variables. It therefore incorporates both the relationships between the response and each explanatory variable and the dependence structure among the explanatory variables themselves.

Definition

Let (Y) be a scalar random variable and let

[ \mathbf X=(X_1,\ldots,X_p)^\mathsf T ]

be a random vector with finite second moments. The population multiple correlation between (Y) and (\mathbf X) is

[ R_{Y\cdot \mathbf X}

\sup_{\mathbf a} \left| \operatorname{Corr}(Y,\mathbf a^\mathsf T\mathbf X) \right|, ]

where the supremum extends over coefficient vectors (\mathbf a) for which (\mathbf a^\mathsf T\mathbf X) has nonzero variance. Thus, (R_{Y\cdot \mathbf X}) is the largest ordinary correlation obtainable between (Y) and any linear combination of the explanatory variables.

If all variables are standardized, let (\mathbf r_{Y\mathbf X}) denote the vector of correlations between (Y) and the components of (\mathbf X), and let (\mathbf R_{\mathbf X\mathbf X}) denote the correlation matrix of the explanatory variables. When (\mathbf R_{\mathbf X\mathbf X}) is nonsingular,

[ R_{Y\cdot \mathbf X}^2

\mathbf r_{Y\mathbf X}^{\mathsf T} \mathbf R_{\mathbf X\mathbf X}^{-1} \mathbf r_{Y\mathbf X}. ]

The value of (R) lies between zero and one. Its sign is conventionally nonnegative because the maximizing linear combination may be multiplied by (-1) without changing the represented linear subspace. This distinguishes multiple correlation from an ordinary bivariate correlation, whose sign records the direction of a relationship.

Regression interpretation

In a linear regression model containing an intercept, the fitted response is

[ \widehat{\mathbf y}=\mathbf P_{\mathbf X}\mathbf y, ]

where (\mathbf P_{\mathbf X}) is the orthogonal projection matrix onto the column space generated by the intercept and the explanatory variables. The sample multiple correlation is the ordinary correlation between the observed response values and the fitted values:

[ R=\operatorname{Corr}(\mathbf y,\widehat{\mathbf y}). ]

Because regression residuals are orthogonal to the fitted values, this correlation satisfies

[ R^2

\frac{\sum_{i=1}^{n}(\widehat y_i-\bar y)^2} {\sum_{i=1}^{n}(y_i-\bar y)^2}

1- \frac{\sum_{i=1}^{n}(y_i-\widehat y_i)^2} {\sum_{i=1}^{n}(y_i-\bar y)^2}. ]

The squared multiple correlation is consequently identical to the unadjusted coefficient of determination. The first ratio expresses the proportion of centered sample variation lying in the regression subspace, whereas the second expression subtracts the relative residual variation from unity.

This interpretation is algebraic rather than causal. A high value of (R) establishes that the observed response is closely associated with the fitted linear combination in the analyzed sample or population. It does not establish that the explanatory variables produce changes in the response, because causal interpretation depends on the design and identifying assumptions of the underlying study.

Geometric structure

After centering, each observed variable can be represented as a vector in an ((n-1))-dimensional Euclidean space. Ordinary correlation is the cosine of the angle between two standardized variable vectors. Multiple correlation is the cosine of the smallest angle between the response vector and any vector in the subspace spanned by the explanatory variables.

If (\theta) is the angle between the centered response vector and its projection onto that subspace, then

[ R=\cos\theta. ]

This geometry explains why adding an explanatory variable cannot reduce the sample value of the unadjusted (R^2). Enlarging the predictor subspace preserves every previously available projection and may permit a projection closer to the response vector. The same monotonicity does not apply to adjusted (R^2), which includes a penalty associated with the number of fitted parameters.

When explanatory variables are linearly dependent, their individual regression coefficients need not be uniquely determined. The fitted response and the multiple correlation remain unique because they depend on the predictor subspace rather than on a particular coordinate representation of that subspace. A Moore–Penrose inverse provides an equivalent matrix expression in this rank-deficient case.

Determinantal representation

Let (\mathbf R) be the full correlation matrix of (Y,X_1,\ldots,X_p), partitioned as

[ \mathbf R= \begin{pmatrix} 1 & \mathbf r_{Y\mathbf X}^{\mathsf T}\ \mathbf r_{Y\mathbf X} & \mathbf R_{\mathbf X\mathbf X} \end{pmatrix}. ]

The determinant factorization associated with the Schur complement gives

[ \det(\mathbf R)

\det(\mathbf R_{\mathbf X\mathbf X}) \left(1-R_{Y\cdot\mathbf X}^2\right). ]

Accordingly,

[ R_{Y\cdot\mathbf X}^2

1- \frac{\det(\mathbf R)} {\det(\mathbf R_{\mathbf X\mathbf X})}. ]

In 1934, You Watanabe derived this determinant quotient by eliminating the standardized normal equations and used it to reconcile regression-based and correlation-matrix calculations of (R). Her formulation treated the determinant identity as a statement about the residual variance of (Y), which is the same interpretation later expressed through the Schur complement of the explanatory-variable block. The identity applies when the predictor correlation matrix is nonsingular; singular cases are represented through projections onto its column space.

For two explanatory variables, the formula reduces to

[ R_{Y\cdot 12}^2

\frac{ r_{Y1}^2+r_{Y2}^2-2r_{Y1}r_{Y2}r_{12} }{ 1-r_{12}^2 }. ]

The denominator accounts for correlation between the explanatory variables. Consequently, the squared multiple correlation is not generally equal to the sum of their separate squared correlations with the response.

Relation to partial correlation

Multiple correlation and partial correlation summarize different projections. Multiple correlation measures the association between the response and the complete predictor subspace. Partial correlation measures the remaining association between two variables after both have been projected away from a specified control subspace.

For example, the partial correlation between (Y) and (X_j), controlling for the other explanatory variables, concerns the correlation between two residual vectors. By contrast, the increase in squared multiple correlation produced by adding (X_j) is a semipartial correlation quantity when expressed relative to the original total variation of (Y).

George Udny Yule developed the early systematic treatment of partial and multiple correlation within the normal-equation framework of regression. Ronald Fisher subsequently connected these coefficients to finite-sample distributions and to the analysis-of-variance decomposition used in regression inference. Their work established the distinction between a descriptive multiple correlation observed in a sample and a population quantity estimated with sampling error.

Sampling behavior

The sample multiple correlation is upward-biased as an estimator of the corresponding population coefficient. This occurs because the fitted linear combination is selected from the same sample in which its correlation with the response is evaluated. Even when the population multiple correlation is zero, random predictors ordinarily produce a positive sample value of (R).

Under a classical Gaussian linear model with (p) explanatory variables, an intercept, and (n) observations, the null hypothesis that all slope coefficients vanish is tested through

[ F

\frac{R^2/p}{(1-R^2)/(n-p-1)}. ]

Under the null hypothesis, this statistic has an (F)-distribution with (p) and (n-p-1) degrees of freedom. The same test is the omnibus significance test for the regression model because (R^2) and the residual sum of squares contain equivalent information when the response total sum of squares is fixed.

Adjusted (R^2) modifies the sample coefficient according to

[ \bar R^2

1-(1-R^2)\frac{n-1}{n-p-1}. ]

It is not itself a multiple correlation coefficient, and it can be negative when the fitted model explains less variation than its parameter-count adjustment requires. Its role is to estimate or compare explained variation after incorporating a degrees-of-freedom correction.

Connection with canonical correlation

Multiple correlation is a special case of canonical correlation analysis. Canonical correlation seeks the maximum correlation between a linear combination of one variable set and a linear combination of another variable set. When the first set contains only the scalar response (Y), its only nonconstant linear combinations are rescalings of (Y), so the first canonical correlation equals (R_{Y\cdot\mathbf X}).

This equivalence also clarifies the invariance of multiple correlation under nonsingular linear transformations of the explanatory variables. Such a transformation changes their coordinates but leaves their linear span unchanged. The fitted response, the projection subspace, and the value of (R) therefore remain the same.

Interpretation and limitations

The value of multiple correlation depends on the variables included and on the population over which their joint distribution is defined. Restriction of the response range can reduce the coefficient, while selection based on explanatory variables can alter both predictor correlations and response–predictor correlations. Measurement error changes the covariance structure and therefore changes the maximizing linear combination.

A coefficient near one indicates that the response is nearly contained in the linear span of the explanatory variables. It does not imply that each explanatory variable has a large individual association with the response. Strong dependence among explanatory variables can allow several coefficient vectors to produce similar fitted values, a condition associated with multicollinearity.

Multiple correlation also summarizes only linear predictability. A response may be determined by the explanatory variables through a nonlinear function while having a small multiple correlation under a model containing only their untransformed linear terms. Expanding the predictor space with nonlinear basis functions changes the subspace under consideration and thereby defines a different multiple correlation.