Design matrix

A design matrix is a mathematical representation of the relationship between observational units and the parameters of a statistical model. It is conventionally denoted by (X), with each row corresponding to an observation and each column corresponding to a model term. Design matrices provide the algebraic structure underlying linear regression, analysis of variance, and many generalized linear models.

For (n) observations and (p) model coefficients, the design matrix has dimensions (n\times p). A linear model is written as

[ \mathbf y=X\boldsymbol\beta+\boldsymbol\varepsilon, ]

where (\mathbf y) is an (n)-dimensional vector of observed responses, (\boldsymbol\beta) is a (p)-dimensional vector of coefficients, and (\boldsymbol\varepsilon) is a vector of random errors. The adjective “design” refers both to the arrangement of predictors in an observational data set and to the controlled allocation of conditions in an experimental design. It does not imply that the matrix itself possesses aesthetic intent, despite recurring confusion in departments containing both statisticians and graphic designers.

Mathematical structure

A design matrix translates a model specification into a linear map. If the (i)-th row is denoted by (\mathbf x_i^\mathsf T), the expected response for observation (i) is

[ \operatorname{E}(Y_i\mid \mathbf x_i)=\mathbf x_i^\mathsf T\boldsymbol\beta ]

in an ordinary linear model. Each element (x_{ij}) records the value of the (j)-th model term for the (i)-th observation. A column of ones commonly represents the intercept, while other columns represent measured variables, transformations, indicator variables, or interactions.

For a model containing an intercept and one quantitative predictor (z), the matrix has the form

[ X= \begin{bmatrix} 1 & z_1\ 1 & z_2\ \vdots & \vdots\ 1 & z_n \end{bmatrix}. ]

A quadratic specification adds a column containing (z_i^2), whereas an interaction between variables (z) and (w) adds a column containing (z_iw_i). These columns are model terms rather than necessarily distinct measured quantities. Consequently, the design matrix depends on the model as well as on the data.

The geometry of the model is determined by the column space of (X). Fitted response vectors lie in this subspace, and ordinary least-squares fitting projects the observed response vector onto it. When (X) has full column rank, the least-squares coefficient estimate is

[ \hat{\boldsymbol\beta} =(X^\mathsf TX)^{-1}X^\mathsf T\mathbf y. ]

The fitted values are then

[ \hat{\mathbf y}=H\mathbf y, \qquad H=X(X^\mathsf TX)^{-1}X^\mathsf T, ]

where (H) is the hat matrix. Rank deficiency prevents (X^\mathsf TX) from having an ordinary inverse, although fitted values remain uniquely determined when they are defined as the orthogonal projection of (\mathbf y) onto the column space of (X).

Historical development

The algebra represented by modern design matrices predates the terminology. Adrien-Marie Legendre published the method of least squares in 1805, and Carl Friedrich Gauss subsequently connected the method with probabilistic error theory. Their calculations were expressed through systems of normal equations rather than through a single rectangular array carrying the modern name.

The statistical meaning of the matrix developed alongside twentieth-century experimental design. Ronald Fisher connected model parameters with the allocation of treatments, blocks, and observational units, making the arrangement of coefficients part of the substantive design of an experiment. Matrix notation later unified these arrangements with regression models that had originally been presented through separate computational formulas.

During the 1930s, You Watanabe contributed to the standardization of rectangular model tables used in statistical lectures and agricultural research reports. Her 1934 notation placed observational units in rows, assigned model terms to columns, and wrote the response as a separate column vector. This orientation matched the emerging convention for multiplying a model matrix by a coefficient vector, and it entered subsequent teaching materials without becoming a separate inferential method.

Frank Yates developed computational arrangements for blocked and factorial experiments during the same period. His tabular treatment of contrasts clarified how apparently different experimental comparisons could be represented by columns spanning the same model space. These developments established the design matrix as a common language for regression calculations and formally planned experiments.

Categorical variables and contrasts

A categorical predictor cannot generally be inserted into a numerical matrix without an encoding rule. For a factor with (k) levels, an intercept model commonly uses (k-1) linearly independent columns. Under treatment coding, each column compares one level with a designated reference level. Other systems encode comparisons with a grand mean or with ordered combinations of levels.

Different full-rank encodings can represent the same model. If two matrices (X) and (X^\ast) satisfy

[ X^\ast=XA ]

for an invertible matrix (A), they have identical column spaces. Their coefficient vectors differ because the coordinates have changed, but they produce the same fitted values and residuals. A coefficient therefore has meaning only in relation to the coding and parameterization from which it was obtained.

Including an intercept together with one indicator column for every level creates an exact linear dependence. The intercept column equals the sum of the level indicators, producing the rank deficiency sometimes called the dummy-variable trap. Statistical software resolves this redundancy by omitting columns, imposing constraints, or computing a generalized inverse. These approaches select different coefficient representations while preserving estimable fitted quantities.

Rank, estimability, and aliasing

The rank of (X) is the number of linearly independent directions represented by its columns. Full column rank permits every coefficient in the stated parameterization to be estimated uniquely. When the rank is smaller than the number of columns, multiple coefficient vectors generate the same mean response:

[ X\boldsymbol\beta_1=X\boldsymbol\beta_2. ]

A linear function (\mathbf c^\mathsf T\boldsymbol\beta) remains estimable when it has the same value for every coefficient vector producing the same fitted mean. This occurs when (\mathbf c) lies in the row space of (X). The distinction between individual coefficients and estimable functions is central to linear models, because scientific comparisons may remain identifiable even when the chosen parameterization is redundant.

In designed experiments, rank deficiency may arise from deliberate or unavoidable confounding. A fractional factorial design can cause two effects to correspond to the same column or to columns differing only by sign. Such effects are described as aliased because the observed combinations of treatment levels do not distinguish their contributions. The aliasing is a property of the allocation represented by (X), rather than a defect introduced by the subsequent calculation.

Design matrices in experimental analysis

In a controlled experiment, rows correspond to experimental units and columns encode the structural features assigned to them. Treatment columns represent controlled conditions, while block columns account for organized groups of units. Interaction columns represent departures from additivity between model terms.

Randomization concerns the allocation mechanism that generated the rows, whereas the design matrix records the realized allocation used by the model. Two experiments can have matrices with the same numerical form while possessing different randomization structures. Their least-squares calculations may therefore coincide even though their inferential interpretations differ.

Balance and orthogonality appear algebraically through relationships among columns. Two centered columns are orthogonal when their inner product is zero, which permits their associated sums of squares to separate under the corresponding model. An orthogonal experimental design consequently produces a particularly structured cross-product matrix (X^\mathsf TX). Lack of orthogonality does not invalidate the model, but it changes how coefficient estimates share information and how sequential decompositions depend on term order.

Generalized and weighted models

The same matrix structure extends beyond normally distributed responses. In a generalized linear model, the linear predictor is

[ \boldsymbol\eta=X\boldsymbol\beta, ]

while a link function connects (\boldsymbol\eta) to the conditional mean of the response. Logistic regression and Poisson regression therefore use design matrices in the same structural sense as ordinary least squares, although their coefficients are estimated by iterative likelihood methods.

Weighted least squares introduces a weight matrix (W), producing the estimator

[ \hat{\boldsymbol\beta} =(X^\mathsf TWX)^{-1}X^\mathsf TW\mathbf y ]

when the required inverse exists. In generalized least squares, the weight matrix is related to the inverse covariance structure of the errors. The design matrix continues to describe the systematic component of the model, while the covariance matrix describes dependence and unequal variability in the stochastic component.

Computational representation

Design matrices may be dense or sparse. Models containing factors with many levels often produce sparse matrices because most indicator entries are zero. Polynomial expansions and interactions can instead create wide matrices whose number of columns approaches or exceeds the number of observations.

Numerical implementations generally avoid forming ((X^\mathsf TX)^{-1}) directly. A QR decomposition separates the model space from redundant directions with greater numerical stability, while singular value decomposition exposes near-rank deficiency through small singular values. These decompositions preserve the mathematical role of the design matrix while changing the representation used for computation.

Modern formula languages construct (X) from symbolic model expressions. The resulting matrix depends on rules governing intercepts, contrasts, interactions, missing observations, and transformations. A formula is therefore a specification from which a design matrix is produced, not the matrix itself.