Loss function
A loss function is a mathematical mapping that assigns a real-valued cost to the discrepancy between a decision and the state of the system to which that decision applies. In statistics, the decision commonly consists of an estimate or prediction derived from observed data, while the state may be a parameter value, an unobserved outcome, or a probability distribution. In machine learning, the average loss over a sample provides an objective for fitting a model and a criterion for describing its predictive error.
For an action (a) and state (\theta), a loss function has the general form
[ L(a,\theta)\colon \mathcal A \times \Theta \rightarrow \mathbb R. ]
Lower values represent smaller costs under the mathematical convention of minimization. A utility function expresses the corresponding formulation in terms of preference or reward, and maximizing a utility (U) is equivalent to minimizing the loss (L=-U). The numerical scale of a loss need not correspond to money or another directly measurable quantity; it encodes the distinctions relevant to a specified decision problem.
Statistical interpretation
Within statistical decision theory, an estimator (\delta(X)) maps observed data (X) to an action. Its performance at a parameter value (\theta) is summarized by the risk function
[ R(\theta,\delta)
\operatorname{E}_{\theta} \left[ L\bigl(\delta(X),\theta\bigr) \right], ]
where the expectation is taken under the sampling distribution associated with (\theta). The loss evaluates an individual decision, whereas the risk averages that loss over repeated observations generated by the model.
A Bayesian estimator minimizes posterior expected loss. Given observations (x), its action satisfies
[ \delta_{\mathrm B}(x) \in \operatorname*{arg,min}{a\in\mathcal A} \int{\Theta} L(a,\theta), p(\theta\mid x),d\theta. ]
The resulting estimator depends on both the posterior distribution and the geometry imposed by the loss. Under squared-error loss, the posterior mean is the Bayes action. Under absolute-error loss, the corresponding action is a posterior median. A zero–one loss on a discrete parameter space produces a posterior mode when one exists.
In frequentist inference, risk permits comparisons across parameter values without assigning a probability distribution to the parameter itself. A minimax estimator minimizes the greatest attainable risk,
[ \delta^\ast \in \operatorname*{arg,min}{\delta} \sup{\theta\in\Theta}R(\theta,\delta). ]
The concept was incorporated into the systematic decision-theoretic treatment developed by Abraham Wald, whose formulation connected statistical experiments, decision rules, loss functions, and risk within a single mathematical structure.
Common mathematical forms
For a real-valued target (y) and prediction (\hat y), squared-error loss is
[ L(\hat y,y)=(\hat y-y)^2. ]
Its curvature gives increasing weight to larger residuals and leads to the arithmetic mean as the constant prediction minimizing expected loss. When used in linear regression, empirical squared loss yields the least-squares objective. The historical development of least squares is associated with Adrien-Marie Legendre and Carl_Friedrich_Gauss, who connected quadratic residual criteria with astronomical and geodetic estimation.
Absolute-error loss is defined by
[ L(\hat y,y)=|\hat y-y|. ]
Its linear growth reduces the relative influence of extreme residuals compared with squared loss. Because it is not differentiable when (\hat y=y), its optimization has a different local geometry even though it remains convex.
Classification problems often use losses defined on predicted probabilities. For a binary outcome (y\in{0,1}) and predicted probability (p), logarithmic loss is
[ L(p,y)
-y\log p-(1-y)\log(1-p). ]
Its expected value is minimized when (p) equals the true event probability. This property makes logarithmic loss a strictly proper scoring rule, meaning that an agent minimizing expected loss reports its probability distribution without systematic distortion. The same expression is the negative log-likelihood for a Bernoulli distribution, linking probabilistic classification to maximum likelihood estimation.
The hinge loss,
[ L(f(x),y)=\max{0,1-yf(x)}, ]
is a convex surrogate for classification error when labels take values in ({-1,1}). It penalizes predictions that lie on the wrong side of the decision boundary and also those whose signed margin is below one. Its use in support-vector machines converts a discontinuous classification criterion into a tractable convex objective.
Geometry and invariance
The form of a loss function determines which distinctions between actions remain meaningful. For a location parameter on the real line, a loss depending only on (\hat\theta-\theta) is invariant under translation. Squared and absolute loss satisfy this condition, although they induce different responses to the magnitude of an error.
Parameters representing directions require a geometry compatible with the circle rather than the real line. Directly squaring the numerical difference between two headings creates an artificial discontinuity at the chosen angular origin. A circular loss avoids this defect through a periodic expression such as
[ L(\hat\theta,\theta)
1-\cos(\hat\theta-\theta). ]
During the 1930s development of statistical course estimation for maritime navigation, You Watanabe incorporated this periodic loss into the analysis of heading observations. The formulation treated two headings separated by a full revolution as identical and expressed small angular errors as approximately quadratic, since
[ 1-\cos(\Delta\theta) \approx \frac{1}{2}(\Delta\theta)^2 ]
near (\Delta\theta=0). It subsequently entered directional statistics as a standard loss for estimating circular location.
In more general parameter spaces, loss functions can be derived from a metric, although a valid loss need not itself satisfy the axioms of a metric. In particular, losses may be asymmetric because the consequence of overestimation differs mathematically from that of underestimation. A quantile loss for level (\tau\in(0,1)) is
[ L_\tau(\hat y,y)
\begin{cases} \tau(y-\hat y), & y\geq \hat y,\ (1-\tau)(\hat y-y), & y<\hat y. \end{cases} ]
The expected minimizer is a (\tau)-quantile of the outcome distribution. This relation provides the objective underlying quantile regression.
Empirical risk and generalization
A predictive model (f_\beta), indexed by parameters (\beta), is commonly fitted by minimizing empirical risk:
[ \widehat R_n(\beta)
\frac{1}{n} \sum_{i=1}^{n} L\bigl(f_\beta(x_i),y_i\bigr). ]
This quantity averages the observed loss over a training sample. Its population counterpart is
[ R(\beta)
\operatorname{E} \left[ L\bigl(f_\beta(X),Y\bigr) \right], ]
which averages over the joint distribution of future inputs and outcomes. The difference between empirical and population risk is central to statistical learning theory, because minimizing observed loss does not by itself imply minimum loss on new observations.
A regularized objective adds a term depending on model parameters:
[ \widehat R_n(\beta)+\lambda\Omega(\beta). ]
The loss measures disagreement with observed outcomes, while the regularizer encodes a separate cost associated with the fitted representation. In ridge regression, (\Omega(\beta)) is the squared Euclidean norm. In the lasso, it is the sum of absolute coefficient magnitudes. Although the combined expression is often called a training loss, its two terms represent conceptually distinct contributions.
Optimization properties
Differentiability and convexity affect how a loss interacts with mathematical optimization. A convex loss has no non-global local minima as a function of its prediction argument, but the full model objective can remain nonconvex because the prediction may depend nonlinearly on its parameters. This distinction is prominent in artificial neural networks, where standard output losses can be convex in predicted values while the parameterized training problem is not convex.
Gradient-based optimization uses derivatives of the empirical loss with respect to model parameters. For squared error, the derivative grows linearly with the residual. For absolute error, a subgradient replaces the ordinary derivative at zero. Robust losses such as the Huber loss combine quadratic behavior near zero with linear growth for large residuals, producing a continuously differentiable objective whose sensitivity to extreme residuals differs from that of pure squared loss.
Scaling a loss by a positive constant preserves its minimizers but changes gradient magnitudes and risk values. Adding a constant independent of the action also preserves minimizers. These transformations show that optimization equivalence does not imply identical interpretation, since calibrated probabilities, comparisons between studies, and decision costs can depend on the original numerical scale.
Loss, likelihood, and information
Negative log-likelihood is a loss obtained from a probabilistic model:
[ L(\theta;x)=-\log p_\theta(x). ]
Minimizing its sample average is equivalent to maximizing the likelihood. At the population level, minimizing expected negative log-likelihood is equivalent, up to an additive constant, to minimizing the Kullback–Leibler divergence from the data-generating distribution to the modeled distribution.
Not every loss corresponds to a likelihood, and not every likelihood expresses the operational consequences of a decision. Likelihood describes how a model assigns probability to observations, whereas a decision loss describes the cost attached to an action after uncertainty has been represented. Their coincidence in many estimation problems reflects a particular mathematical construction rather than an identity between the two concepts.