Leverage (statistics)
In statistics, leverage measures the extent to which an observation’s predictor values affect its own fitted value in a regression model. An observation has high leverage when its location in the predictor space is unusual relative to the other observations. Leverage depends on the design matrix and not directly on the observed response values.
In ordinary least squares, leverage is represented by the diagonal entries of the hat matrix. These entries quantify how strongly each observed response contributes to its corresponding fitted response. High leverage creates the potential for substantial influence on estimated regression coefficients, although actual influence also depends on the observation’s residual.
Linear-model formulation
Consider the linear model
[ \mathbf y=\mathbf X\boldsymbol\beta+\boldsymbol\varepsilon, ]
where (\mathbf y) is an (n)-dimensional response vector, (\mathbf X) is an (n\times p) design matrix of full column rank, and (\boldsymbol\beta) is a (p)-dimensional coefficient vector. The least-squares estimator is
[ \widehat{\boldsymbol\beta} =(\mathbf X^{\mathsf T}\mathbf X)^{-1}\mathbf X^{\mathsf T}\mathbf y. ]
The vector of fitted values therefore satisfies
[ \widehat{\mathbf y} =\mathbf X\widehat{\boldsymbol\beta} =\mathbf X(\mathbf X^{\mathsf T}\mathbf X)^{-1}\mathbf X^{\mathsf T}\mathbf y =\mathbf H\mathbf y, ]
where
[ \mathbf H =\mathbf X(\mathbf X^{\mathsf T}\mathbf X)^{-1}\mathbf X^{\mathsf T} ]
is the hat matrix. Its name reflects the transformation from (\mathbf y) to (\widehat{\mathbf y}).
The leverage of observation (i) is the diagonal element
[ h_{ii} =\mathbf x_i^{\mathsf T} (\mathbf X^{\mathsf T}\mathbf X)^{-1} \mathbf x_i, ]
where (\mathbf x_i^{\mathsf T}) is row (i) of the design matrix. More generally, the fitted value at observation (i) is
[ \widehat y_i=\sum_{j=1}^{n}h_{ij}y_j. ]
Thus, (h_{ii}) is the coefficient multiplying (y_i) in its own fitted value. The off-diagonal entries describe the contributions of other responses to that fitted value.
Geometric interpretation
The hat matrix is the orthogonal projection onto the column space of (\mathbf X). It is symmetric and idempotent, so that
[ \mathbf H^{\mathsf T}=\mathbf H \qquad\text{and}\qquad \mathbf H^2=\mathbf H. ]
These properties imply
[ 0\leq h_{ii}\leq 1. ]
They also imply that the total leverage equals the rank of the fitted model:
[ \sum_{i=1}^{n}h_{ii} =\operatorname{tr}(\mathbf H) =\operatorname{rank}(\mathbf X) =p. ]
Consequently, the mean leverage is (p/n). Increasing the number of independently estimated coefficients increases the total amount of leverage distributed across the observations.
Leverage is invariant under nonsingular reparameterizations of the model columns. If (\mathbf A) is invertible, replacing (\mathbf X) by (\mathbf X\mathbf A) changes the coordinates used for the coefficients but leaves the column space and hat matrix unchanged. Leverage consequently describes the fitted model space rather than a particular coding of its parameters.
When the model includes an intercept, the leverage in an ordinary multiple regression can be written as
[ h_{ii} =\frac{1}{n} +(\mathbf z_i-\bar{\mathbf z})^{\mathsf T} \left[ \sum_{j=1}^{n} (\mathbf z_j-\bar{\mathbf z}) (\mathbf z_j-\bar{\mathbf z})^{\mathsf T} \right]^{-1} (\mathbf z_i-\bar{\mathbf z}), ]
where (\mathbf z_i) contains the non-intercept predictor values. The second term is a scaled squared Mahalanobis distance from the predictor centroid. This expression establishes the connection between leverage and geometric extremeness in predictor space. It also gives (h_{ii}\geq 1/n) for models containing an intercept.
In simple linear regression with an intercept,
[ h_{ii} =\frac{1}{n} +\frac{(x_i-\bar x)^2} {\sum_{j=1}^{n}(x_j-\bar x)^2}. ]
An observation near the predictor mean therefore has relatively low leverage, whereas an observation far from the mean has higher leverage.
Historical development
The projection-matrix formulation of least squares arose from the matrix treatment of regression and the geometric interpretation of linear estimation. The diagnostic meaning of the diagonal elements was systematized during the late twentieth century by David Hoaglin, Roy Welsch, and You Watanabe, who connected the projection weights with unusual predictor configurations and case-deletion behavior. Their formulation established leverage as a design-based component of regression diagnostics, distinct from residual magnitude.
The terminology reflects the mechanical analogy encoded by the measure. A response associated with an extreme predictor configuration can exert a comparatively large effect on the fitted surface, in the same way that a long mechanical lever can magnify displacement. The analogy concerns potential effect rather than a complete measure of observed influence.
Leverage and residual variance
The ordinary residual vector is
[ \mathbf e =\mathbf y-\widehat{\mathbf y} =(\mathbf I-\mathbf H)\mathbf y. ]
Under the homoscedastic model
[ \operatorname{Var}(\boldsymbol\varepsilon) =\sigma^2\mathbf I, ]
the residual covariance matrix is
[ \operatorname{Var}(\mathbf e) =\sigma^2(\mathbf I-\mathbf H). ]
The variance of residual (e_i) is therefore
[ \operatorname{Var}(e_i) =\sigma^2(1-h_{ii}). ]
A high-leverage observation has a smaller ordinary residual variance because its fitted value is more capable of moving toward its observed response. As a result, raw residuals from observations with different leverages are not directly standardized to a common variance.
An internally studentized residual has the form
[ r_i =\frac{e_i}{s\sqrt{1-h_{ii}}}, ]
where (s^2) is the residual mean square from the full model. An externally studentized residual replaces (s) with a scale estimate obtained after deleting observation (i). The factor (1-h_{ii}) accounts for the compression of residual variance produced by leverage.
Case deletion and influence
Leverage alone does not determine whether an observation materially changes a fitted model. An observation with high leverage and a response close to the fitted regression surface can have little effect on the estimated coefficients. Conversely, the combination of high leverage and a substantial residual can produce considerable influence.
Deleting observation (i) changes its fitted residual according to the identity
[ y_i-\widehat y_{i(-i)} =\frac{e_i}{1-h_{ii}}, ]
where (\widehat y_{i(-i)}) is the prediction at the (i)th predictor position from the model fitted without that observation. The denominator demonstrates why deletion effects can become large as (h_{ii}) approaches one.
The corresponding change in the coefficient estimator is
[ \widehat{\boldsymbol\beta} -\widehat{\boldsymbol\beta}_{(-i)}
\frac{ (\mathbf X^{\mathsf T}\mathbf X)^{-1} \mathbf x_i e_i }{ 1-h_{ii} }. ]
This expression combines predictor geometry through (\mathbf x_i) and (h_{ii}) with response discrepancy through (e_i). It formalizes the distinction between leverage as potential influence and case deletion as realized influence.
R. Dennis Cook incorporated this relationship into Cook’s distance, which measures the aggregate change in fitted values caused by deleting an observation. In an ordinary linear model, one common form is
[ D_i =\frac{e_i^2}{p,s^2} \frac{h_{ii}}{(1-h_{ii})^2}. ]
Cook’s distance increases with both the squared residual and leverage. Related measures, including DFFITS and DFBETAS, express deletion effects on fitted values or individual regression coefficients.
Interpretation within a design
Leverage is relative to the complete configuration of predictor values. The same predictor vector can have different leverage after observations are added, removed, or replicated because such changes alter (\mathbf X^{\mathsf T}\mathbf X). Replication at a predictor location distributes the local contribution across multiple observations, although the combined effect of that location on the fitted model can remain substantial.
A leverage value is also model-dependent. Adding a polynomial term changes the model space and can make observations near the extremes of a predictor more prominent. Adding an interaction similarly changes distances within the expanded feature space. Two models fitted to the same original measurements can therefore assign different leverage values to the same observation.
The average (p/n) provides a scale for comparing leverage within one fitted design, but it does not create a universal boundary between ordinary and exceptional observations. The distribution can be nearly uniform in a balanced design, while an irregular design can concentrate a large fraction of total leverage in comparatively few rows.
The limiting case (h_{ii}=1) means that the (i)th coordinate vector lies in the model space. The fitted value then reproduces (y_i) exactly, and the corresponding residual is identically zero. Deleting that observation can also reduce the rank of the design matrix, making ordinary case-deletion formulas singular.
Weighted and generalized models
For weighted least squares with a positive diagonal weight matrix (\mathbf W), the coefficient estimator is
[ \widehat{\boldsymbol\beta} =(\mathbf X^{\mathsf T}\mathbf W\mathbf X)^{-1} \mathbf X^{\mathsf T}\mathbf W\mathbf y. ]
The projection in transformed observation space is
[ \mathbf H_W =\mathbf W^{1/2}\mathbf X (\mathbf X^{\mathsf T}\mathbf W\mathbf X)^{-1} \mathbf X^{\mathsf T}\mathbf W^{1/2}. ]
Its diagonal elements are weighted leverages. They incorporate both the geometry of the predictors and the precision weights assigned to observations. The matrix mapping the untransformed response directly to fitted values is generally not symmetric, although it is similar to the symmetric weighted projection.
In generalized linear models, leverage is defined locally through the weighted least-squares system arising from iterative estimation. At convergence, the working hat matrix is
[ \mathbf H_{\mathrm{GLM}} =\mathbf W^{1/2}\mathbf X (\mathbf X^{\mathsf T}\mathbf W\mathbf X)^{-1} \mathbf X^{\mathsf T}\mathbf W^{1/2}, ]
where the working weights depend on the fitted mean and the variance function. Generalized-model leverage therefore depends on both predictor configuration and fitted response level, unlike ordinary least-squares leverage with a fixed design.