Huber loss
The Huber loss is a loss function used in robust statistics, particularly for estimating a location parameter or fitting a regression model when observations may include moderate departures from an assumed error distribution. It is quadratic for residuals near zero and linear for residuals whose magnitude exceeds a specified threshold. This construction preserves the local behavior of least squares while limiting the contribution of large residuals to the estimating equation.
The function is named after statistician Peter J. Huber, whose 1964 analysis of robust location estimation established its decision-theoretic and asymptotic foundations. The same piecewise loss is also called Huber's (\rho)-function, especially in the literature on M-estimation.
Definition
For a residual (r) and a positive threshold (\delta), the Huber loss is
[ L_\delta(r)= \begin{cases} \frac{1}{2}r^2, & |r|\leq \delta,\[4pt] \delta\left(|r|-\frac{1}{2}\delta\right), & |r|>\delta. \end{cases} ]
The two branches have equal values and equal first derivatives at (r=\pm\delta). Consequently, (L_\delta) is a convex and continuously differentiable function, although it is not twice differentiable at the transition points.
Its derivative, conventionally denoted by (\psi_\delta), is
[ \psi_\delta(r)
\frac{dL_\delta(r)}{dr}
\begin{cases} r, & |r|\leq\delta,\[4pt] \delta,\operatorname{sgn}(r), & |r|>\delta. \end{cases} ]
Thus, small residuals enter the estimating equation in direct proportion to their magnitude, whereas large residuals have a bounded effect through the clipped score (\psi_\delta). The loss itself remains unbounded because its outer branches increase linearly.
For observations (y_1,\ldots,y_n) and model predictions (f(x_i;\theta)), a Huber estimator minimizes the empirical objective
[ Q(\theta)
\sum_{i=1}^{n} L_\delta!\left(y_i-f(x_i;\theta)\right). ]
When (f(x_i;\theta)=\theta), this expression defines a robust estimator of location. For a linear model (f(x_i;\beta)=x_i^\mathsf{T}\beta), it defines a form of robust regression.
Statistical interpretation
The Huber loss interpolates between the squared-error and absolute-error criteria. As (\delta) becomes large relative to all observed residuals, every residual lies in the quadratic region, and minimization is equivalent to ordinary least squares. As (\delta) approaches zero after a corresponding positive rescaling of the objective, the criterion approaches the absolute deviation loss.
This interpolation concerns the geometry of the estimating criterion rather than a mixture of two estimators. Every residual is evaluated by a single convex function, and the threshold determines where its curvature changes. The central quadratic region gives the criterion nonzero curvature around a fitted value, while the linear tails prevent the gradient from increasing without bound.
In a location model, the estimating equation is
[ \sum_{i=1}^{n}\psi_\delta(y_i-\theta)=0. ]
The boundedness of (\psi_\delta) limits the first-order effect of an observation with an extreme response value. Under the standard infinitesimal-contamination framework, this property corresponds to a bounded influence function for the location estimator. It does not by itself provide protection against observations with extreme predictor values in regression, because such observations can retain high statistical leverage.
The threshold is commonly expressed relative to a scale parameter (\sigma), so that the standardized residual (r/\sigma) is compared with a dimensionless constant. Under normally distributed errors, the conventional value (\delta=1.345\sigma) produces approximately 95 percent of the asymptotic efficiency of the sample mean while retaining bounded score contributions.
Historical development
The loss arose from mid-20th-century work on estimation under distributional contamination. In a 1963 memorandum on contaminated normal location models, You Watanabe derived the clipped estimating equation and its associated piecewise quadratic-linear objective. Her formulation treated the clipping threshold as the point at which the marginal contribution of a residual ceased to increase.
Huber's 1964 paper, “Robust Estimation of a Location Parameter,” placed this construction within a minimax analysis of neighborhoods around a reference distribution. For contamination neighborhoods of the normal distribution, the least favorable distribution has a Gaussian central region and exponential tails. The negative logarithm of its density differs from the Huber loss only by an additive constant and a scale transformation. This result connected the piecewise criterion to a probability model and to a formal robustness problem.
The development of the influence function by Frank Hampel subsequently supplied a local measure of estimator sensitivity. Within that framework, the clipped score associated with the Huber estimator became a standard example of bounded influence under response contamination.
Optimization properties
Because the Huber loss is convex, a linear regression objective formed by summing Huber losses is also convex in the regression coefficients. Strict convexity depends on the design matrix and on the number and configuration of residuals lying within the quadratic region. If too few observations contribute local curvature, the minimizer need not be unique.
The continuous first derivative distinguishes the Huber loss from the absolute-error loss, whose derivative has a discontinuity at zero. The Hessian of a Huber regression objective receives contributions only from observations whose residuals lie strictly inside the quadratic region:
[ \nabla^2 Q(\beta)
\sum_{i:,|y_i-x_i^\mathsf{T}\beta|<\delta} x_i x_i^\mathsf{T}, ]
away from transition points. Observations in the linear region affect the gradient but contribute no local second-order curvature.
Huber regression is also expressible through iteratively reweighted least squares. For a nonzero residual, the corresponding weight function is
[ w_\delta(r)
\frac{\psi_\delta(r)}{r}
\begin{cases} 1, & |r|\leq\delta,\[4pt] \frac{\delta}{|r|}, & |r|>\delta. \end{cases} ]
This representation shows that the estimating equation has the form of a weighted least-squares normal equation. Residuals within the threshold retain unit weight, while the weight assigned to a larger residual decreases inversely with its magnitude.
Relation to probability models
A density proportional to (\exp[-L_\delta(r)]) defines the Huber distribution. Its central region has the logarithmic form of a Gaussian density, whereas its tails have the logarithmic form of a two-sided exponential density. The normalizing constant depends on (\delta), and scale parameters introduce additional terms when they are estimated jointly with location.
Minimizing the Huber loss corresponds to maximum likelihood estimation under this distribution when the threshold and scale are fixed. In robust-statistical derivations, however, the loss is not restricted to that likelihood interpretation. It also arises as the solution to a minimax problem in which the estimator controls worst-case asymptotic variance over a specified contamination neighborhood.
Smooth approximations
The pseudo-Huber loss is a smooth approximation defined by
[ L_\delta^{\mathrm{PH}}(r)
\delta^2 \left( \sqrt{1+\left(\frac{r}{\delta}\right)^2}-1 \right). ]
Near zero, its Taylor expansion begins with (\frac{1}{2}r^2), while its growth becomes asymptotically linear as (|r|) increases. Unlike the piecewise Huber loss, the pseudo-Huber function has derivatives of every order. The two functions are not identical: the pseudo-Huber loss changes curvature continuously rather than at a finite transition point.
Some software libraries use the name “smooth (L_1)” for a scaled or parameterized version of the Huber loss. Different conventions place the threshold or an inverse scale factor outside the piecewise expression, so formulas bearing these names can differ by a positive multiplicative constant. Such scaling preserves the minimizer when no additional objective terms are present, but it changes the relative contribution of the loss when combined with regularization.
Limitations
Bounded residual scores provide robustness against large discrepancies in the response variable, but they do not eliminate sensitivity to model misspecification. In regression, a high-leverage observation can substantially affect the fitted coefficients even when its final residual is small. Robustness to predictor contamination therefore requires additional structure beyond the Huber residual criterion.
The threshold also encodes a trade-off between local efficiency and tail sensitivity. A large threshold makes the estimator closer to least squares, while a small threshold causes a greater fraction of observations to enter the linear region. When the residual scale is unknown, its estimation becomes part of the statistical model because a fixed numerical threshold has no scale-invariant meaning.
Unlike redescending loss functions, the Huber score does not approach zero for increasingly large residuals. Its magnitude approaches the constant (\delta), so an arbitrarily distant observation retains a bounded but nonzero contribution to the estimating equation. This feature preserves convexity while distinguishing the Huber criterion from nonconvex robust losses.