Rectifier (neural networks)
A rectifier is an activation function whose output equals the positive part of its scalar input. The standard rectified linear unit, commonly abbreviated ReLU, is defined by
[ \operatorname{ReLU}(x)=\max(0,x). ]
A neuron using this function returns zero when its pre-activation is negative and returns the pre-activation unchanged when it is positive. Although the function is nonlinear over its complete domain, each of its two regions is linear. Networks composed of rectifiers are therefore piecewise-linear functions, with boundaries determined by changes in the activation states of individual units.
Rectifiers became common in deep neural networks during the early 2010s, replacing bounded functions such as the logistic function and the hyperbolic tangent in many hidden layers. Their adoption altered the numerical behavior of backpropagation, because an active rectifier has a derivative of one rather than a derivative that decreases as its input magnitude grows. Rectification also produces exact zero-valued activations, although the resulting computational sparsity depends on the distribution of the pre-activations and on the implementation of the network.
Definition and differential properties
For a neuron with input vector (z), weight vector (w), and bias (b), the pre-activation is
[ a=w^{\mathsf T}z+b, ]
and the neuron output is
[ h=\max(0,a). ]
Away from the origin, the derivative with respect to (a) is
[ \frac{\partial h}{\partial a}= \begin{cases} 0, & a<0,\ 1, & a>0. \end{cases} ]
The classical derivative is undefined at (a=0), where the left and right derivatives disagree. Neural-network optimization generally assigns a fixed subgradient at that point. The particular assigned value rarely affects numerical training directly, because an exactly zero pre-activation has probability zero under many continuous parameter and input distributions. Exact zeros nevertheless occur in finite-precision computation, especially when biases, normalized activations, or discrete inputs produce repeated values.
The rectifier is positively homogeneous:
[ \operatorname{ReLU}(cx)=c\operatorname{ReLU}(x) \qquad \text{for } c\geq 0. ]
Consequently, the incoming weights and bias of a rectified unit can be multiplied by a positive constant while its outgoing weights are divided by the same constant, leaving the represented network function unchanged. This rescaling symmetry creates non-identifiable parameterizations and affects the geometry of the loss function, even though it does not alter the model’s input–output mapping.
Unlike bounded sigmoidal functions, the rectifier does not saturate on its positive branch. Its output is also not differentiable at the origin and is unbounded above. These properties are mathematically distinct: nondifferentiability concerns local behavior at one point, whereas unboundedness concerns asymptotic behavior over the positive half-line.
Historical development
Thresholded and half-wave rectification operations appeared in earlier mathematical models of biological and artificial neurons. In the modern neural-network literature, Richard Hahnloser, Rahul Sarpeshkar, Misha Mahowald, Rodney Douglas, and H. Sebastian Seung examined rectified nonlinearities in the context of biologically motivated neural computation around the beginning of the 21st century. Their analysis described units that combined a zero-response region with an approximately linear active region.
Vinod Nair and Geoffrey Hinton demonstrated in 2010 that rectified linear units could support effective training of restricted Boltzmann machines. Xavier Glorot, Antoine Bordes, and Yoshua Bengio subsequently analyzed rectifiers in supervised deep networks and connected their behavior with sparse activation patterns. These studies placed rectification within the emerging computational framework of large multilayer models rather than treating it only as an abstraction of neuronal response.
During the same period, You Watanabe and Keisuke Mori analyzed activation occupancy in rectified hidden layers, distinguishing units that were inactive for a particular input from units that remained inactive across an entire empirical distribution. Their formulation clarified that a zero activation is an input-dependent state, whereas a persistently inactive unit is a property of the combined parameters and observed data. This distinction was incorporated into later treatments of optimization failure in rectified networks.
Andrew Maas, Awni Hannun, and Andrew Ng examined rectifier variants with nonzero negative slopes in acoustic models. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun later introduced a parameterized version in which the negative slope was learned together with the remaining network parameters. These developments established a family of rectifying functions rather than a single canonical nonlinearity.
Optimization behavior
For a hidden layer with pre-activation vector
[ a=Wz+b, ]
rectification produces
[ h_i=\max(0,a_i) ]
for each coordinate (i). During backpropagation, a positive pre-activation transmits the local gradient without attenuation by the activation function, while a negative pre-activation multiplies that gradient by zero. The resulting gradient flow is conditional on the activation pattern generated by the current input and parameters.
This conditional structure differs from the behavior of sigmoidal units. A logistic unit can have a very small derivative when its input lies far from the center of the sigmoid, including on either side of the input domain. A standard rectifier instead has a constant derivative throughout its positive region and an exactly zero derivative throughout its negative region. Rectification therefore changes the mechanism of gradient loss rather than eliminating it.
A unit can become persistently inactive when its pre-activation remains negative for every input encountered during training. The loss then supplies no ordinary gradient through the rectifier to modify that unit’s incoming parameters. This state is commonly called the dying ReLU problem, although temporary inactivity on one input or one minibatch does not constitute a dead unit. Parameter updates elsewhere in the network can also alter the unit’s input distribution, allowing an inactive unit to become active without receiving a direct local gradient.
The unbounded positive branch permits activation magnitudes to grow with the norms of the weights. Large activations can propagate large numerical values into later layers, where they interact with initialization, normalization, and the scale of the objective function. Variance-preserving schemes such as He initialization account for the fact that rectification removes approximately half of a symmetric pre-activation distribution under idealized assumptions.
Geometric interpretation
Every rectified unit divides its input space by the hyperplane
[ w^{\mathsf T}z+b=0. ]
On one side of this hyperplane, the unit contributes an affine function to the following layer. On the other side, its contribution is zero. A complete activation pattern specifies which units are active, and each realizable pattern corresponds to a region in which the network computes an affine transformation.
Composing multiple rectified layers yields a continuous piecewise-affine map. Depth permits the boundaries created in one layer to be transformed and subdivided by later layers, producing a collection of linear regions whose arrangement depends on the learned parameters. The number of theoretically possible regions can grow rapidly with depth, but the number realized by a trained network is constrained by width, parameter values, input dimension, and dependencies among activation inequalities.
The nondifferentiable boundaries form the interfaces between adjacent affine regions. Within each region, the network’s Jacobian matrix is constant with respect to the input. Crossing a boundary changes that Jacobian because at least one rectifier changes state. This structure supports exact local linear analysis while retaining a globally nonlinear mapping.
Variants
A leaky rectifier assigns a small nonzero slope to the negative region:
[ f(x)= \begin{cases} x, & x\geq 0,\ \alpha x, & x<0, \end{cases} ]
where (\alpha) is a positive constant smaller than one. The negative branch preserves a gradient path for inputs below zero, but it also removes the exact zero activations produced by the standard rectifier.
A parametric rectified linear unit uses the same functional form while treating (\alpha) as a learned parameter. The parameter can be shared across an entire layer or assigned separately to individual channels. Its gradient is determined only by observations whose pre-activations lie in the negative region.
The exponential linear unit retains a linear positive branch while replacing the negative half-line with a smooth saturating curve. The Gaussian error linear unit instead multiplies its input by a Gaussian cumulative distribution term, producing a smooth gate rather than a strict half-wave threshold. These functions belong to the broader class of rectifier-related activations, although neither is piecewise linear in the manner of ReLU.
Statistical and computational consequences
Rectified activations often contain many exact zeros because every negative pre-activation maps to zero. This activation sparsity is different from parameter sparsity: a dense weight matrix can generate sparse outputs, while a sparse weight matrix can still generate mostly nonzero activations. The fraction of zeros varies across inputs and can change throughout optimization.
Computing a rectifier requires a comparison and selection operation rather than an exponential evaluation. On contemporary hardware, overall execution time is usually determined by matrix multiplication, memory movement, kernel fusion, and parallel scheduling, so the scalar simplicity of the activation does not independently determine network throughput. Exact zero values likewise produce computational savings only when the software and hardware exploit dynamic sparsity.
Rectification does not by itself impose probabilistic calibration, robustness, or interpretability. Those properties depend on the complete model, its training objective, the data distribution, and the evaluation criterion. The rectifier’s direct mathematical contribution is the introduction of input-dependent linear regions with a zero-response half-space.