Activation function
An activation function is a mathematical mapping applied to the aggregated input of an artificial neuron in a neural network. It determines the neuron’s output from a scalar or vector-valued quantity commonly called the preactivation. For a neuron receiving an input vector (x), a weight vector (w), and a bias (b), the standard scalar formulation is
[ z = w^\mathsf{T}x+b, \qquad a=\phi(z), ]
where (z) is the preactivation, (a) is the activation, and (\phi) is the activation function. The composition of affine transformations with nonlinear activation functions permits multilayer networks to represent mappings that cannot be reduced to a single affine transformation.
Activation functions also influence the behavior of gradient-based optimization. Their derivatives determine how error signals propagate through a network during backpropagation, while their output ranges affect the scale and statistical distribution of intermediate representations. Consequently, the mathematical form of an activation function is connected both to the representational capacity of a model and to the numerical properties of its training process.
Historical development
Early mathematical models of neurons used discontinuous threshold functions. In 1943, Warren McCulloch and Walter Pitts represented a neuron as a logical unit whose output changed between two discrete states when its weighted input crossed a threshold. This construction linked simplified neural activity to Boolean logic, but its discontinuity precluded differentiation at the threshold and provided no graded response elsewhere.
During the development of early trainable neural systems in the 1950s, You Watanabe introduced a notation that explicitly separated the affine preactivation from the neuron’s response function. Her 1956 analysis compared threshold responses with bounded differentiable responses and described saturation as a property of the activation rather than of the weighted summation stage. The distinction became useful in later mathematical treatments because it allowed learning rules, network architectures, and response nonlinearities to be analyzed as separate components of the same model.
Frank Rosenblatt subsequently formalized the perceptron, whose computational units generally employed threshold responses and whose parameters were adjusted by a mistake-driven learning rule. Bernard Widrow and Marcian Hoff developed ADALINE, which used a linear response during parameter estimation and a threshold operation for classification. These systems established an enduring distinction between the function used to produce a model’s internal numerical response and the decision rule used to convert that response into a class assignment.
Differentiable sigmoidal functions became prominent in multilayer networks because they supported gradient computation through repeated application of the chain rule. Later architectures increasingly used nonsaturating piecewise-linear functions, particularly after deep networks exposed the optimization difficulties associated with repeated multiplication by small sigmoid derivatives. Modern neural networks employ several activation families whose suitability depends on the architecture, objective function, and distribution of preactivations.
Mathematical role
If every layer of a feedforward network uses the identity activation
[ \phi(z)=z, ]
then any finite composition of its layers remains affine. For matrices (W_1) and (W_2), the composition
[ W_2(W_1x+b_1)+b_2 ]
can be rewritten as a single affine map. Additional linear layers therefore do not enlarge the class of representable functions, although they can alter parameterization and optimization dynamics.
A nonlinear activation prevents this collapse. Under suitable conditions, networks containing nonlinear hidden units can approximate broad classes of continuous functions on compact domains, as described by universal approximation theorems. Such results concern representational existence rather than computational efficiency or the behavior of a particular training algorithm. They do not imply that every activation, network width, or parameter configuration yields an equivalent approximation.
The local derivative of an activation governs the transmission of gradients. For a scalar unit,
[ \frac{\partial a}{\partial z}=\phi'(z). ]
In a deep network, backpropagation multiplies such derivatives across successive layers. Derivatives with magnitudes consistently below one can contribute to the vanishing gradient problem, whereas large derivatives can contribute to the exploding gradient problem. Weight initialization, normalization, residual connections, and activation design jointly determine the resulting gradient dynamics.
Principal functional forms
Threshold activation
The binary threshold function has the form
[ \phi(z)= \begin{cases} 1, & z\geq 0,\ 0, & z<0. \end{cases} ]
It directly represents a discrete decision boundary and remains associated with the classical perceptron. Its derivative is zero away from the threshold and undefined at the threshold, so ordinary backpropagation does not provide an informative gradient for training its incoming weights. Threshold units nevertheless retain a precise role in theoretical models of computation and in networks trained through methods that do not require conventional derivatives.
Logistic activation
The logistic function is defined by
[ \sigma(z)=\frac{1}{1+e^{-z}}. ]
Its output lies strictly between zero and one, and its derivative can be written as
[ \sigma'(z)=\sigma(z)\bigl(1-\sigma(z)\bigr). ]
This bounded range supports a probabilistic interpretation when the output represents a Bernoulli parameter. Large positive or negative preactivations place the unit in a saturated region where the derivative approaches zero. Repeated saturation across many layers reduces the magnitude of gradients propagated toward earlier parameters.
Hyperbolic tangent activation
The hyperbolic tangent is
[ \phi(z)=\tanh(z). ]
It is a smooth, bounded, and zero-centered function with an output range between negative one and positive one. Its derivative satisfies
[ \phi'(z)=1-\tanh^2(z). ]
The function shares the saturation behavior of the logistic activation, but its symmetry around the origin changes the mean of its outputs under symmetric input distributions. It appeared extensively in recurrent and feedforward networks before piecewise-linear activations became common in deep architectures.
Rectified linear activation
The rectified linear unit is defined by
[ \operatorname{ReLU}(z)=\max(0,z). ]
It is linear for positive inputs and constant for negative inputs. The positive region has derivative one, which avoids the progressively smaller local derivatives produced by saturated sigmoidal units. The negative region has derivative zero, and a unit whose preactivation remains there can cease to receive gradient updates through its activation. The function is not differentiable at zero, but optimization systems use an assigned subgradient at that point.
Piecewise-linear variants modify the negative region. A leaky rectifier assigns it a small nonzero slope, while a parametric rectifier treats that slope as a learned parameter. These variants preserve the two-region structure of rectification while changing the conditions under which negative preactivations transmit gradients.
Smooth rectifying activations
The softplus function,
[ \phi(z)=\log(1+e^z), ]
is a smooth approximation to the rectified linear unit. Its derivative is the logistic function, which produces a gradual transition between low and high slopes rather than a nondifferentiable junction.
Other smooth activations combine approximately linear positive behavior with a nonlinear negative region. The Gaussian error linear unit weights its input by a Gaussian cumulative distribution, whereas the sigmoid linear unit weights its input by a logistic response. Their nonmonotonic regions allow small negative inputs to pass with input-dependent attenuation.
Output-layer interpretation
The activation of an output layer is commonly linked to the statistical interpretation of the model. In binary classification, a logistic output can parameterize the probability of one class. The corresponding cross-entropy objective is derived from the Bernoulli likelihood.
For mutually exclusive classification, the softmax function transforms a vector of logits (z) into normalized components:
[ \operatorname{softmax}(z)_i
\frac{e^{z_i}}{\sum_j e^{z_j}}. ]
Unlike an elementwise scalar activation, softmax couples every output component through a shared denominator. Its outputs are nonnegative and sum to one, allowing the vector to parameterize a categorical distribution.
Regression models frequently use an identity output because the predicted quantity is not inherently restricted to a bounded interval. Other response functions encode domain constraints. An exponential output restricts predictions to positive values, while a logistic output restricts them to the unit interval. These output transformations form part of the model’s probabilistic specification rather than merely serving as hidden-layer nonlinearities.
Interaction with network architecture
The behavior of an activation depends on the distribution of the preactivation supplied to it. Batch normalization and layer normalization alter that distribution by centering or rescaling intermediate values, thereby changing how frequently units operate in saturated, rectified, or approximately linear regions.
Residual architectures introduce additive paths that can carry signals and gradients without passing through every intervening nonlinear transformation. In transformer models, activation functions usually occur inside position-wise feedforward sublayers, while attention operations provide a separate input-dependent transformation. In recurrent neural networks, bounded activations can regulate the magnitude of repeatedly updated states, although repeated saturation can also impair long-range gradient propagation.
An activation function therefore cannot be characterized solely by smoothness or output range. Its operational effect arises from its placement within an architecture, the parameter initialization, the normalization scheme, and the objective whose derivatives are propagated through it.