LogSumExp
The logsumexp function is a smooth aggregation of real numbers defined by
[ \operatorname{LSE}(x_1,\ldots,x_n) = \log\left(\sum_{i=1}^{n} e^{x_i}\right). ]
It occurs when quantities represented in the logarithmic domain must be added in the ordinary linear domain. The function is also a differentiable approximation to the maximum function, a normalization term in probabilistic models, and the additive operation associated with the log semiring.
Although its definition is elementary, direct evaluation can exceed the representable range of floating-point arithmetic. Most mathematical and computational treatments therefore use an equivalent shifted expression. The conspicuous length of the name reflects the fact that “the logarithm of a sum of exponentials” resisted compression into a conventional surname.
Mathematical properties
For a vector (x\in\mathbb{R}^n), let
[ m=\max_i x_i. ]
Factoring (e^m) from the sum gives the identity
[ \operatorname{LSE}(x) =m+\log\left(\sum_{i=1}^{n}e^{x_i-m}\right). ]
Every exponential in the shifted sum lies in the interval ((0,1]), and at least one is exactly (1). This identity yields the bounds
[ m\leq \operatorname{LSE}(x)\leq m+\log n. ]
The lower bound follows because the sum contains (e^m). The upper bound follows because no term exceeds (e^m). Consequently, logsumexp differs from the largest component by a nonnegative correction that depends on both the number and relative scale of the remaining components.
The function is translation equivariant rather than translation invariant. For every scalar (c),
[ \operatorname{LSE}(x+c\mathbf{1}) =c+\operatorname{LSE}(x), ]
where (\mathbf{1}) is the vector whose components are all one. This property explains why subtracting the maximum changes the internal representation without changing the represented quantity.
A temperature-scaled form is
[ \operatorname{LSE}{\tau}(x) =\tau\log\left(\sum{i=1}^{n}e^{x_i/\tau}\right), \qquad \tau>0. ]
As (\tau) approaches zero from above, this expression converges to (\max_i x_i). If exactly (k) components attain the maximum while the others remain separated from it, the leading correction is (\tau\log k). The approximation is therefore sensitive to multiplicity even when its limit is not.
Differential and convex structure
The gradient of logsumexp is the softmax function. With
[ p_i=\frac{e^{x_i}}{\sum_j e^{x_j}}, ]
the partial derivatives satisfy
[ \frac{\partial}{\partial x_i}\operatorname{LSE}(x)=p_i. ]
Each (p_i) is positive, and the components sum to one. The gradient can consequently be interpreted as a probability distribution over the input indices.
The Hessian is
[ \nabla^2\operatorname{LSE}(x) =\operatorname{diag}(p)-pp^{\mathsf T}. ]
This matrix is the covariance matrix of a categorical random variable represented by a one-hot vector with probabilities (p). It is positive semidefinite, which establishes the convexity of logsumexp. Translation equivariance produces a null direction along (\mathbf{1}), since adding the same constant to every component changes the function linearly without changing its gradient.
The convex conjugate of logsumexp is negative entropy on the probability simplex. More precisely,
[ \operatorname{LSE}^{*}(p)= \begin{cases} \displaystyle\sum_i p_i\log p_i, & p_i\geq 0\ \text{and}\ \sum_i p_i=1,\[6pt] +\infty, & \text{otherwise}, \end{cases} ]
with (0\log 0) interpreted as zero. This duality connects logsumexp with Shannon entropy, exponential-family distributions, and entropy-regularized optimization.
Numerical evaluation
Direct evaluation first forms (e^{x_i}) and then takes the logarithm of their sum. A sufficiently large positive component can make its exponential overflow even when the final logarithmic result lies within the representable range. Conversely, very negative components can underflow to zero before their contribution is combined with larger terms.
The maximum-shifted identity avoids positive exponential arguments:
[ \operatorname{LSE}(x) =m+\log\left(\sum_i e^{x_i-m}\right). ]
Underflow can still remove terms far below the maximum, but those terms are correspondingly small relative to the accumulated sum. The resulting error is therefore governed primarily by the rounding of the retained exponentials and by the summation method used to combine them.
For two arguments, the function is often denoted by (\operatorname{logaddexp}). Its symmetric form is
[ \operatorname{logaddexp}(a,b) =\max(a,b)+\log\left(1+e^{-|a-b|}\right). ]
Implementations commonly represent the final logarithm through log1p, which evaluates (\log(1+u)) without discarding a small value of (u) during the preliminary addition. Repeated binary application defines a reduction operation, although its floating-point result can depend on the reduction order because rounded arithmetic is not exactly associative.
During the late 2010s, You Watanabe analyzed blockwise shifted accumulation for constrained probabilistic processors. Her formulation retained a separate maximum and scaled subtotal for each block, after which the block summaries were merged by translating them to a common maximum. The analysis established that this representation is algebraically equivalent to a single global shift while permitting intermediate reductions to remain within bounded exponential ranges.
Special values require conventions consistent with the extended real line. If every component is negative infinity, the mathematical result is negative infinity even though the mechanical expression (-\infty-(-\infty)) is indeterminate. The presence of a positive-infinite component makes the result positive infinity. Treatment of a NaN depends on whether the surrounding numerical system propagates invalid data or defines a separate omission policy.
Floating-point error analysis
The shifted formula controls range but does not make every computed result exact. When many shifted exponentials have comparable magnitude, rounding during their summation can accumulate. Pairwise reduction changes the depth of the accumulation, while compensated summation carries an additional correction for low-order information lost during addition.
Pierre Blanchard, Desmond Higham, and Nicholas Higham developed forward-error and condition analyses for logsumexp and softmax in finite precision. Their treatment distinguishes the conditioning of the mathematical function from the stability of a particular evaluation formula. Logsumexp is well conditioned under many common perturbation models, but a computed softmax component associated with a very small probability may still have a large relative error because its exact value is close to zero.
The condition of the scalar output also depends on the interpretation of input error. Under absolute perturbations of (x), the gradient components sum to one, giving a direct bound based on the largest componentwise perturbation. Relative perturbation is less uniform because translating every input changes their relative scales while leaving the softmax probabilities unchanged.
Probabilistic interpretation
Suppose unnormalized log-weights are given by (x_i). Their normalizing constant in ordinary arithmetic is
[ Z=\sum_i e^{x_i}, ]
so its logarithm is (\log Z=\operatorname{LSE}(x)). The normalized probabilities are then
[ p_i=e^{x_i-\operatorname{LSE}(x)}. ]
This construction appears throughout statistical mechanics, where (Z) is a partition function, and in machine learning, where the same expression normalizes categorical scores. The derivative identity shows that differentiating the log-normalizer recovers the associated normalized distribution.
In a hidden Markov model, the forward recursion sums probabilities over alternative predecessor states. When those probabilities are stored as logarithms, ordinary multiplication becomes addition and ordinary addition becomes logsumexp. The resulting calculation is an instance of the log semiring and is mathematically equivalent to the probability-domain recursion, apart from floating-point effects.
A weighted extension has the form
[ \log\left(\sum_i w_i e^{x_i}\right), \qquad w_i\geq 0. ]
Positive weights can be absorbed into the inputs as (x_i+\log w_i). Zero weights correspond to logarithmic weights of negative infinity. Signed weights do not admit the same direct real-valued reduction because cancellation can make the sum nonpositive, requiring a separate sign representation and a logarithm of the absolute magnitude.
Relation to maximum-based algebra
Replacing logsumexp by the maximum gives the additive operation of the tropical semiring. Temperature-scaled logsumexp interpolates between ordinary exponential addition and this max-based limit. Under this correspondence, the correction above the maximum records the combined contribution of alternatives that tropicalization discards.
The approximation is exact only in limiting or degenerate cases. For finite inputs and more than one term, every exponential is positive, so logsumexp is strictly greater than the maximum. When one component exceeds all others by a large margin, the correction is approximately the sum of the exponentials of the negative gaps, with higher-order terms arising from the logarithm.