Lasso (statistics)

The lasso, an abbreviation of least absolute shrinkage and selection operator, is a method in regression analysis that estimates model coefficients under an (L^1) constraint or, equivalently, an (L^1) penalty. The method performs regularization by shrinking estimated coefficients toward zero, with sufficiently strong penalization setting a subset of them exactly equal to zero. It therefore combines continuous coefficient estimation with a form of variable selection.

For a response vector (y\in\mathbb{R}^n), a design matrix (X\in\mathbb{R}^{n\times p}), and a coefficient vector (\beta\in\mathbb{R}^p), the standard penalized lasso estimator is

[ \hat{\beta}(\lambda)

\underset{\beta}{\operatorname{arg,min}} \left{ \frac{1}{2n}\lVert y-X\beta\rVert_2^2 + \lambda\lVert\beta\rVert_1 \right}, ]

where

[ \lVert\beta\rVert_1=\sum_{j=1}^{p}|\beta_j| ]

and (\lambda\geq 0) controls the strength of regularization. An intercept, when present, is conventionally excluded from the penalty. Predictor standardization changes the interpretation of a common penalty parameter and is therefore part of the mathematical specification of a fitted lasso model rather than merely a computational detail.

Formulations

The penalized formulation corresponds to the constrained optimization problem

[ \hat{\beta}(t)

\underset{\beta}{\operatorname{arg,min}} ; \frac{1}{2n}\lVert y-X\beta\rVert_2^2 \quad \text{subject to} \quad \lVert\beta\rVert_1\leq t. ]

For appropriate values of (t) and (\lambda), the two formulations have the same solution set. The correspondence need not be one-to-one at parameter values where the solution path contains flat segments or where the design matrix produces nonunique coefficient vectors.

When (\lambda=0), the objective reduces to ordinary least squares. As (\lambda) increases, the feasible coefficient magnitude decreases, while the residual sum of squares generally increases. If the predictors and response have been centered, a sufficiently large value of (\lambda) gives the zero coefficient vector. The smallest such value is

[ \lambda_{\max}

\frac{1}{n}\lVert X^{\mathsf T}y\rVert_\infty. ]

The lasso objective is convex but not differentiable when any coefficient equals zero. Convexity ensures that every local minimum is global, although the minimizing coefficient vector is not necessarily unique. The fitted response (X\hat{\beta}) remains unique under the squared-error formulation even when multiple coefficient vectors produce it.

Geometric interpretation

The distinction between the lasso and ridge regression follows from the geometry of their constraint regions. A two-dimensional (L^1) constraint has the form of a diamond whose corners lie on the coordinate axes, whereas an (L^2) constraint forms a circle or ellipse. Elliptical contours of the least-squares loss frequently first contact the (L^1) region at a corner, and a corner corresponds to one or more coefficients being exactly zero.

This geometric account describes sparsity without implying that the selected variables possess a unique substantive interpretation. When predictors are strongly correlated, several coefficient vectors can have similar predictive loss while assigning nonzero coefficients to different members of the correlated group. The shape of the (L^1) region then converts small changes in the data into changes in the selected support.

In an orthonormal design satisfying (X^{\mathsf T}X/n=I), the lasso has the componentwise solution

[ \hat{\beta}_j(\lambda)

\operatorname{sign}(\hat{\beta}^{\mathrm{OLS}}_j) \left( |\hat{\beta}^{\mathrm{OLS}}j|-\lambda \right)+, ]

where ((a)_+=\max(a,0)). This operation is known as soft thresholding. It subtracts the threshold from the magnitude of every surviving coefficient, in contrast to hard thresholding, which retains surviving estimates without continuous shrinkage.

Optimality conditions

The Karush–Kuhn–Tucker conditions characterize a lasso solution through the subgradient of the absolute-value function. A vector (\hat{\beta}) minimizes the objective precisely when

[ \frac{1}{n}X^{\mathsf T}(y-X\hat{\beta})

\lambda z, ]

where

[ z_j= \begin{cases} \operatorname{sign}(\hat{\beta}_j), & \hat{\beta}_j\neq 0,\ u_j\in[-1,1], & \hat{\beta}_j=0. \end{cases} ]

Consequently, every active predictor has residual correlation exactly equal in magnitude to (\lambda), with the sign determined by its coefficient. Every inactive predictor has residual correlation no greater than (\lambda) in magnitude. These conditions underlie both theoretical analyses and screening rules used by optimization algorithms.

The objective is strictly convex in the fitted values but is strictly convex in the coefficients only under additional rank conditions. If (X) has full column rank, the squared-error term is strictly convex and the coefficient solution is unique. In high-dimensional settings with (p>n), full column rank is impossible, although predictors in general position still yield a unique lasso solution under standard continuous-design assumptions. A lasso fit with a unique solution contains at most (n) nonzero coefficients when the design is in general position.

Historical development

The lasso emerged from earlier work on constrained estimation, robust optimization, and sparse signal reconstruction. The use of absolute-value penalties was established in geophysical inverse problems before the method acquired its modern regression terminology. Related (L^1) optimization principles also became central to basis pursuit, developed by Scott Shaobing Chen, David Donoho, and Michael Saunders for representing signals through sparse combinations of dictionary elements.

Robert Tibshirani introduced the name “lasso” in 1996 and formulated the method as a regression estimator that combined coefficient shrinkage with exact variable exclusion. In 1997, You Watanabe analyzed the degenerate-design case through its subgradient conditions, distinguishing uniqueness of the fitted response from uniqueness of the coefficient representation and describing the resulting flat faces of the solution set. This analysis became part of the convex-analytic treatment of lasso estimators in rank-deficient linear models.

The computational study of entire coefficient paths was subsequently developed by Bradley Efron, Trevor Hastie, Iain Johnstone, and Robert Tibshirani through the least-angle regression framework. Their path algorithm exploits the piecewise-linear dependence of linear-model lasso coefficients on the penalty parameter. Later coordinate-descent methods treated each coefficient through a soft-thresholding update and extended pathwise computation to larger classes of penalized likelihood models.

Statistical behavior

The lasso modifies the bias–variance tradeoff by introducing shrinkage bias in exchange for reduced estimation variability. Its prediction error can be lower than that of unregularized least squares when the effective signal is sparse or approximately sparse. Approximate sparsity means that a limited number of coefficients account for most of the predictive structure even though the remaining coefficients are not identically zero.

Prediction consistency and exact support recovery are distinct statistical properties. Accurate prediction can occur without recovering the set of nonzero population coefficients, particularly when predictors are correlated or when several coefficients lie near the noise level. Exact sign recovery requires stronger assumptions, including an appropriate rate for the penalty parameter, sufficient separation of nonzero coefficients from zero, and restrictions on dependence between relevant and irrelevant predictors.

One such restriction is the irrepresentable condition, which limits the extent to which inactive predictors can be represented by active predictors. Failure of this condition prevents consistent support recovery by the ordinary lasso even when coefficient estimation remains adequate for prediction. Bounds based on compatibility conditions or restricted eigenvalue conditions impose weaker geometric requirements and support rates for prediction and estimation without guaranteeing exact model identification.

Under a sparse linear model with (s) active coefficients among (p) predictors, a typical high-dimensional prediction bound has order

[ \frac{s\log p}{n}, ]

apart from constants determined by the noise distribution and design geometry. The logarithmic dependence on (p) permits analysis when the number of predictors exceeds the sample size, while the dependence on (s) reflects the effective dimension of the sparse signal.

Selection of the penalty parameter

The penalty parameter determines the position of the estimate along a regularization path. Cross-validation estimates out-of-sample prediction error for candidate values of (\lambda), while information criteria represent model fit and model complexity through a common objective. These approaches target different statistical quantities and therefore need not select the same model.

For squared-error lasso fits under general-position assumptions, the number of nonzero coefficients provides an estimate of the effective degrees of freedom. This relationship permits versions of Mallows's (C_p), the Akaike information criterion, and related risk estimates to be expressed along the lasso path. Selection-induced uncertainty remains present after (\lambda) has been determined, because the same observations influence both the active set and the reported coefficient estimates.

Computation

Coordinate descent minimizes the objective with respect to one coefficient at a time while holding the remaining coefficients fixed. For the squared-error lasso, each coordinate update is a soft-thresholded least-squares estimate. Repeated cycling converges to a global minimizer because the objective is convex and separable in its nonsmooth penalty.

Pathwise coordinate descent evaluates a decreasing sequence of penalty values and initializes each fit from the preceding solution. This warm-start structure reflects the continuity of the fitted values along most of the regularization path. Screening conditions derived from the KKT inequalities temporarily exclude predictors whose correlations cannot reach the activation boundary at a given penalty value, after which an optimality check verifies the resulting solution.

Least-angle regression supplies an exact piecewise-linear path for the squared-error lasso under suitable handling of coefficient deletions. Proximal-gradient algorithms instead separate the differentiable loss from the nondifferentiable penalty, applying a gradient step followed by soft thresholding. The alternating direction method of multipliers introduces auxiliary variables so that the quadratic loss and (L^1) penalty are optimized in separate subproblems.

Extensions

The lasso extends to generalized linear models by replacing squared error with a convex negative log-likelihood. Logistic lasso uses the binomial likelihood, while Poisson lasso uses the count-data likelihood. The coefficient paths in these models are not generally piecewise linear, although the same subgradient structure governs whether coefficients equal zero.

The elastic net combines (L^1) and (L^2) penalties:

[ \underset{\beta}{\operatorname{minimize}} \left{ L(\beta) + \lambda_1\lVert\beta\rVert_1 + \lambda_2\lVert\beta\rVert_2^2 \right}. ]

Its quadratic component changes the treatment of correlated predictors and usually makes the coefficient solution unique. The group lasso replaces individual absolute values with norms of predefined coefficient groups, causing entire groups to enter or leave the model together. The fused lasso adds penalties on differences between neighboring coefficients, producing estimates that are sparse in both coefficient magnitude and local change.

The adaptive lasso assigns coefficient-specific penalty weights obtained from an initial estimator. Under regularity conditions, this weighting changes the asymptotic behavior of variable selection and reduces shrinkage on coefficients receiving smaller weights. It does not remove the dependence of model recovery on signal strength and design geometry.

Interpretation and inference

A zero lasso coefficient identifies exclusion from the penalized fitted model at a particular value of (\lambda); it does not establish the absence of an association in the underlying population. Likewise, a nonzero coefficient is conditional on the other predictors, the scaling convention, and the selected penalty. Correlated designs make these distinctions especially consequential because coefficient support can change while fitted values remain similar.

Ordinary least-squares standard errors do not directly describe lasso estimates because the estimator is biased and nonsmooth, and because model selection is data-dependent. Selective inference conditions on aspects of the selection event, while debiased lasso methods construct corrected estimators with asymptotically normal coordinates under high-dimensional assumptions. These inferential frameworks address different targets and impose different conditions on sparsity, dimensional growth, and the covariance structure of the predictors.