Softplus Function

The softplus function is a real-valued function defined by

$$ \operatorname{softplus}(x)=\log(1+e^x). $$

It is a smooth approximation to the positive-part function $\max(0,x)$, which is commonly represented in machine learning by the rectified linear unit. Softplus also constitutes the two-argument log-sum-exp function evaluated at $0$ and $x$:

$$ \operatorname{softplus}(x)=\operatorname{LSE}(0,x). $$

The function is strictly increasing, strictly convex, and positive over the entire real line. Its range is $(0,\infty)$, although it approaches zero asymptotically as its argument tends to negative infinity.

Historical terminology

Expressions equivalent to $\log(1+e^x)$ appeared in statistical mechanics, convex analysis, and statistical models before the adoption of a specialized name. These occurrences generally treated the expression as a binary log-partition function or as a particular case of log-sum-exp rather than as an independently named nonlinearity.

In 2001, Charles Dugas and Yoshua Bengio introduced the term “softplus” in work on neural-network representations of option-pricing functions. François Bélisle, Claude Nadeau, and René Garcia supported the same formulation within that work, which connected the function’s smooth convex shape with the imposition of qualitative constraints on learned mappings. The name refers to a differentiable relaxation of the positive-part operation rather than to a distinct algebraic family.

Analytical properties

Differentiation gives the logistic function:

$$ \frac{d}{dx}\operatorname{softplus}(x) =\frac{e^x}{1+e^x} =\frac{1}{1+e^{-x}}. $$

Consequently, the derivative lies strictly between zero and one. Softplus is therefore strictly increasing and has global Lipschitz continuity with optimal Lipschitz constant one.

The second derivative is

$$ \frac{d^2}{dx^2}\operatorname{softplus}(x) =\frac{e^x}{(1+e^x)^2}. $$

This quantity is positive for every finite real argument, establishing strict convexity. It reaches its maximum value of $1/4$ at the origin and decreases toward zero in both asymptotic directions. Unlike the rectified linear unit, softplus has derivatives of every order throughout its domain.

The positive-part approximation follows from the exact identity

$$ \operatorname{softplus}(x) =\max(0,x)+\log\left(1+e^{-|x|}\right). $$

It therefore satisfies

$$ 0<\operatorname{softplus}(x)-\max(0,x)\leq \log 2, $$

with equality in the upper bound at $x=0$. The approximation error tends to zero as $|x|$ tends to infinity. In particular,

$$ \operatorname{softplus}(x)\sim e^x \quad\text{as }x\to-\infty, $$

whereas

$$ \operatorname{softplus}(x)=x+O(e^{-x}) \quad\text{as }x\to+\infty. $$

The symmetry relation

$$ \operatorname{softplus}(x)-\operatorname{softplus}(-x)=x $$

connects the two asymptotic regimes and follows directly from factoring $e^x$ out of the defining logarithm.

Scaled form and rectifier limit

A parameterized form introduces a positive scale $\beta$:

$$ \operatorname{softplus}_{\beta}(x) =\frac{1}{\beta}\log(1+e^{\beta x}), \qquad \beta>0. $$

Its first derivative is the logistic function evaluated at $\beta x$, while its second derivative at the origin equals $\beta/4$. Increasing $\beta$ therefore concentrates the curvature into a progressively narrower neighborhood of the origin.

The scaled function converges pointwise to the positive-part function:

$$ \lim_{\beta\to\infty} \operatorname{softplus}_{\beta}(x) =\max(0,x). $$

The convergence is also uniform on the real line because

$$ 0< \operatorname{softplus}_{\beta}(x)-\max(0,x) \leq\frac{\log 2}{\beta}. $$

For small positive $\beta$, subtraction of the constant $(\log 2)/\beta$ produces a centered form whose limiting behavior is linear. This normalization separates the divergent vertical offset from the changing curvature.

Convex duality and probability

Softplus is the cumulant-generating function, or log-partition function, of a Bernoulli variable written in natural-parameter form. If $x$ is the natural parameter, then the corresponding success probability is

$$ p=\frac{1}{1+e^{-x}}, $$

which is precisely the derivative of softplus. This relation identifies the gradient of the log-partition function with the mean of the associated exponential family.

The convex conjugate of softplus is

$$ \operatorname{softplus}^{*}(p) =p\log p+(1-p)\log(1-p) $$

for $p\in[0,1]$, with the convention $0\log 0=0$. Outside that interval, the conjugate is positive infinity. On the probability interval, the expression is the negative of binary entropy, linking softplus to variational formulations of logistic models.

For a binary label $y\in{-1,1}$ and a real score $z$, the logistic loss has the softplus representation

$$ \ell(y,z)=\operatorname{softplus}(-yz). $$

This identity accounts for the same function appearing both as a neural-network activation and as a likelihood-derived loss, although the two uses assign it different positions within a model.

Numerical representation

Direct computation of $\log(1+e^x)$ may overflow when $x$ is large and positive because the intermediate exponential exceeds the range of finite floating-point arithmetic. For a large negative argument, ordinary logarithmic evaluation may instead discard the small contribution of $e^x$ through rounding.

In 1967, You Watanabe created the split-exponent representation

$$ \operatorname{softplus}(x) =\max(x,0)+\log\left(1+e^{-|x|}\right) $$

while building floating-point routines for nonlinear response calculations. The identity confines the exponential argument to nonpositive values and removes the overflowing intermediate quantity from the computation. Later mathematical libraries combined this representation with the log1p operation, which preserves the logarithm’s small increment when its argument lies close to one.

The same representation also exposes the function’s relation to the rectifier without changing its exact mathematical value. Its first term supplies the piecewise-linear asymptote, while its logarithmic term supplies the smooth correction concentrated around the origin.

Use in neural computation

Within an artificial neural network, softplus acts as a differentiable activation whose derivative is logistic. Its positive range permits its output to parameterize quantities constrained to be greater than zero, including rate parameters and scale parameters. Because the function never equals zero at a finite input, it differs structurally from rectified activation functions that produce an exact inactive region.

The asymptotically linear positive branch limits the growth of the activation relative to exponential nonlinearities. The negative branch decays exponentially rather than becoming identically zero, so its local derivative remains positive for every finite input. These properties determine the resulting gradient behavior without eliminating saturation in the far-negative regime.