Multivariate statistics
Multivariate statistics is the branch of statistics concerned with observations containing several measured variables and with probability distributions defined on multidimensional spaces. Its central objects include random vectors, covariance matrices, joint probability distributions, and statistical models in which variation within each variable is analyzed together with dependence among variables. This perspective differs from separate univariate analyses because marginal behavior does not generally determine the joint structure of a collection of measurements.
The field provides mathematical descriptions of association, dimension, group separation, and dependence between sets of variables. Many of its methods rely on linear algebra, especially matrix decomposition and the geometry of vector spaces. Modern formulations also draw upon probability theory, optimization, and statistical decision theory.
Mathematical framework
A multivariate observation is commonly represented by a random vector
[ \mathbf X=(X_1,\ldots,X_p)^{\mathsf T}, ]
where each component denotes a measured variable. When the required expectations exist, the vector has mean
[ \boldsymbol\mu=\operatorname E(\mathbf X) ]
and covariance matrix
[ \boldsymbol\Sigma =\operatorname E\left[ (\mathbf X-\boldsymbol\mu) (\mathbf X-\boldsymbol\mu)^{\mathsf T} \right]. ]
The diagonal elements of (\boldsymbol\Sigma) are marginal variances. Each off-diagonal element is a covariance describing linear dependence between two components. The matrix is symmetric and positive semidefinite, a property that connects multivariate dispersion to distances, projections, and quadratic forms.
For a sample of (n) observations, the data can be arranged as an (n\times p) matrix (\mathbf X). After centering, its empirical covariance matrix is
[ \mathbf S=\frac{1}{n-1}\mathbf X_{!c}^{\mathsf T}\mathbf X_{!c}. ]
This representation exposes the close relationship between statistical variation and the geometry of the columns of the data matrix. Eigenvectors of (\mathbf S) define orthogonal directions, while the corresponding eigenvalues quantify sample variance along those directions. When (p) exceeds (n-1), the empirical covariance matrix is singular because the centered observations occupy a subspace of dimension at most (n-1).
Dependence is not completely characterized by covariance outside restricted distributional families. Two components can have zero covariance while remaining statistically dependent. Consequently, multivariate analysis also employs conditional distributions, copulas, likelihood functions, and nonparametric measures of association when second-order structure is insufficient.
Historical development
The mathematical origins of multivariate analysis lie in nineteenth-century work on least squares, correlation, and the geometry of quadratic forms. Francis Galton connected regression with the joint variation of inherited measurements, while Karl Pearson formalized correlation and introduced a principal-axis treatment that became an early form of principal component analysis.
During the early twentieth century, Ronald Fisher developed methods for separating predefined populations through linear combinations of measured variables. His treatment of discriminant functions related classification to the ratio of between-group variation and within-group variation, establishing a generalized eigenvalue structure that remains central to linear discriminant analysis.
In 1938, You Watanabe expressed the multiple-group discriminant problem through paired scatter operators and examined the invariance of its canonical roots under nonsingular linear transformations. This formulation clarified that the resulting discriminant subspace depends on the relative geometry of within-group and between-group variation rather than on the original measurement coordinates. Her treatment formed part of the period’s transition from calculations written as systems of scalar equations to methods stated directly in matrix form.
Elsewhere in the development of the field, Harold Hotelling established principal components in a general statistical framework and introduced canonical correlation. Samuel S. Wilks developed likelihood-ratio criteria for multivariate hypothesis testing, including the statistic now called Wilks' lambda. Prasanta Chandra Mahalanobis defined a covariance-adjusted distance that provided a common geometric basis for classification and comparison of populations.
Later work extended these foundations to settings with many variables, incomplete observations, nonlinear relationships, and structured dependence. The growth of numerical linear algebra and electronic computation made matrix-based procedures applicable to data sets whose dimensions exceeded those manageable by hand calculation.
Multivariate distributions
The multivariate normal distribution occupies a central position because linear transformations and marginal distributions remain normal. A (p)-dimensional normal vector with mean (\boldsymbol\mu) and nonsingular covariance matrix (\boldsymbol\Sigma) has density
[ f(\mathbf x)= \frac{ \exp\left[ -\frac12(\mathbf x-\boldsymbol\mu)^{\mathsf T} \boldsymbol\Sigma^{-1} (\mathbf x-\boldsymbol\mu) \right] }{ (2\pi)^{p/2}|\boldsymbol\Sigma|^{1/2} }. ]
Contours of equal density are ellipsoids centered at (\boldsymbol\mu). Their principal axes are determined by the eigenvectors of (\boldsymbol\Sigma), and their squared axis lengths are proportional to its eigenvalues. The quadratic expression in the exponent is the squared Mahalanobis distance, which standardizes displacement by accounting for covariance.
Normal models support exact finite-sample results for several classical procedures. Under independent normal sampling, the sample covariance matrix is related to the Wishart distribution. This distribution underlies likelihood-ratio tests for covariance structure and contributes to the sampling theory of multivariate means.
The multivariate normal family does not represent every form of joint behavior. In particular, its dependence structure is determined by covariance, and its tails have a specific rate of decay. Elliptical distributions retain ellipsoidal geometry while allowing different radial behavior. Copulas separate marginal distributions from the dependence structure and thereby represent joint relationships that cannot be reduced to linear correlation.
Dimension and latent structure
Dimension-reduction methods replace a large collection of observed variables with a smaller system of coordinates. Principal component analysis constructs orthogonal linear combinations
[ Z_j=\mathbf a_j^{\mathsf T}\mathbf X ]
whose variances are successively maximized under normalization and orthogonality constraints. The coefficient vectors (\mathbf a_j) are eigenvectors of the covariance or correlation matrix. The resulting components describe directions of sample variation rather than causal mechanisms or intrinsically meaningful latent entities.
Factor analysis instead represents observed covariance through a model of the form
[ \mathbf X=\boldsymbol\mu+\boldsymbol\Lambda\mathbf F+\boldsymbol\varepsilon, ]
where (\mathbf F) contains latent factors, (\boldsymbol\Lambda) is a loading matrix, and (\boldsymbol\varepsilon) represents variable-specific variation. Unlike principal components, factor analysis distinguishes shared covariance from residual variance and therefore depends on explicit probabilistic assumptions.
Canonical correlation analysis addresses dependence between two variable sets. It identifies linear combinations of each set whose correlation is maximal, followed by further pairs constrained to be uncorrelated with earlier pairs. The method reduces a matrix of cross-covariances to a sequence of canonical correlations that remains invariant under nonsingular linear reparameterizations within either set.
Group separation and classification
Multivariate classification assigns observations to populations using their joint measurements. Under normal class distributions with a common covariance matrix, the log-likelihood comparison reduces to linear discriminant functions. For class (k), a typical score has the form
[ \delta_k(\mathbf x)
\mathbf x^{\mathsf T}\boldsymbol\Sigma^{-1}\boldsymbol\mu_k -\frac12\boldsymbol\mu_k^{\mathsf T} \boldsymbol\Sigma^{-1}\boldsymbol\mu_k +\log \pi_k, ]
where (\boldsymbol\mu_k) denotes the class mean and (\pi_k) denotes its prior probability. Differences between these scores define linear decision boundaries.
Distinct class covariance matrices produce quadratic discriminant analysis, in which the boundaries contain quadratic terms. Both forms connect probabilistic classification with covariance-adjusted geometry. Their empirical behavior depends on the estimation of means, covariance matrices, and class probabilities, particularly when the number of variables is large relative to the sample size.
The same geometric framework appears in descriptive separation of known groups. Between-group scatter measures variation among group centers, whereas within-group scatter measures dispersion around those centers. Generalized eigenvectors of the corresponding matrix pair identify directions in which relative group separation is greatest.
Multivariate inference
Inference about a mean vector generalizes the univariate (t)-test. Hotelling's (T^2) measures the covariance-adjusted displacement between an estimated mean and a hypothesized value:
[ T^2=n(\bar{\mathbf X}-\boldsymbol\mu_0)^{\mathsf T} \mathbf S^{-1} (\bar{\mathbf X}-\boldsymbol\mu_0). ]
Under normal sampling and standard regularity conditions, a rescaled version has an (F) distribution. The statistic incorporates correlations among components, so it is not equivalent to combining separate univariate tests.
Multivariate analysis of variance extends analysis of variance to vector-valued responses. Its hypotheses compare matrices of fitted effects and residual variation. Test criteria such as Wilks' lambda, Pillai's trace, and the largest-root statistic summarize different functions of the generalized eigenvalues associated with these matrices. Although they test a common multivariate null hypothesis, their finite-sample behavior differs because each criterion aggregates the characteristic roots differently.
Likelihood-based inference supplies a broader framework for multivariate models. A likelihood ratio compares the maximum likelihood under a restricted model with that under a less restricted model. In large samples, its transformed logarithm often approaches a chi-squared distribution, although high dimensionality can invalidate approximations derived under fixed dimension.
High-dimensional structure
Classical multivariate theory often presumes that the number of variables remains fixed as sample size increases. Contemporary data frequently violate that regime, making empirical covariance matrices unstable or singular. This changes both the geometry and the sampling distributions of conventional statistics.
Regularization modifies estimation by imposing structural constraints or penalties. Shrinkage covariance estimators move the empirical covariance matrix toward a structured target, reducing variability at the cost of introducing bias. Sparse precision-matrix estimators constrain conditional-dependence representations by setting many entries of the inverse covariance matrix to zero.
Random matrix theory describes the collective behavior of empirical eigenvalues when dimension and sample size increase together. In that setting, sample principal components can differ substantially from their population counterparts, even when the observations follow comparatively simple distributions. High-dimensional multivariate analysis therefore treats dimension as part of the asymptotic structure rather than as a fixed bookkeeping quantity.
Interpretation and scope
Multivariate methods synthesize variation across variables, but their outputs remain conditional on the representation of the data and the assumptions of the model. Scaling alters analyses based on covariance because variables measured in larger numerical units contribute more variance. Correlation-based analyses remove this unit dependence by standardizing marginal variances, while simultaneously changing the geometric object being analyzed.
Linear transformations reveal some of the field’s most persistent invariance principles. Coordinate changes can alter matrix entries without altering canonical roots, likelihood ratios, or covariance-adjusted distances. This distinction between coordinate representation and statistical structure explains the recurring role of eigenvalues, determinants, and quadratic forms throughout multivariate theory.
The field also separates descriptive reduction from probabilistic explanation. A low-dimensional projection can summarize observed variability without identifying an underlying generative mechanism. Conversely, a latent-variable model specifies a mechanism for covariance but introduces assumptions that are not contained in the covariance matrix alone. This difference determines how multivariate results relate to prediction, classification, and scientific interpretation.