Exploding gradient problem
The exploding gradient problem is a numerical and dynamical instability that arises during the training of artificial neural networks, particularly networks whose computations extend across many sequential layers or time steps. The problem occurs when repeated applications of the chain rule produce derivatives whose magnitudes grow exponentially with computational depth. The resulting gradients can cause disproportionately large parameter updates, unstable optimization trajectories, arithmetic overflow, and failure of the training objective to converge.
Exploding gradients are closely related to the vanishing gradient problem. Both phenomena originate in products of Jacobian matrices, but they correspond to different growth regimes. Vanishing gradients arise when these products contract perturbations, whereas exploding gradients arise when they amplify them. A single model can exhibit both regimes in different parameter directions because the relevant Jacobians generally have nonuniform singular-value spectra.
Mathematical formulation
Consider a recurrent state governed by
[ \mathbf{h}t = f(\mathbf{W}\mathbf{h}{t-1}+\mathbf{U}\mathbf{x}_t+\mathbf{b}), ]
where (\mathbf{h}_t) is the hidden state, (\mathbf{x}_t) is the input, (\mathbf{W}) is the recurrent weight matrix, and (f) is a differentiable activation function. If a loss (L) at time (T) depends on an earlier state (\mathbf{h}_t), its derivative contains the product
[ \frac{\partial L}{\partial \mathbf{h}_t}
\frac{\partial L}{\partial \mathbf{h}T} \prod{k=t+1}^{T} \frac{\partial \mathbf{h}k}{\partial \mathbf{h}{k-1}}. ]
Each factor is a Jacobian matrix of the form
[ \frac{\partial \mathbf{h}k}{\partial \mathbf{h}{k-1}}
\mathbf{D}_k\mathbf{W}, ]
where (\mathbf{D}_k) contains derivatives of the activation function evaluated along the state trajectory. The norm of the complete product is bounded by
[ \left| \prod_{k=t+1}^{T}\mathbf{D}k\mathbf{W} \right| \leq \prod{k=t+1}^{T} \left|\mathbf{D}_k\mathbf{W}\right|. ]
When the characteristic amplification represented by these factors remains greater than one over a sufficiently long interval, the product grows approximately exponentially with (T-t). The gradient with respect to (\mathbf{W}) contains a sum of related products and can therefore become extremely large even when the forward states remain finite.
The eigenvalues of (\mathbf{W}) provide an informative approximation when the Jacobians vary little or commute, but the general behavior depends on singular values, matrix non-normality, and the trajectory-dependent derivatives in (\mathbf{D}_k). Consequently, a spectral radius below one does not by itself exclude transient gradient growth. Non-normal matrices can produce substantial finite-time amplification even when their eigenvalues indicate asymptotic contraction.
Dynamical interpretation
A recurrent network defines a parameterized dynamical system. Backpropagated gradients describe how perturbations of an earlier state influence a later loss, so exploding gradients correspond to rapidly increasing sensitivity along particular directions in state space. This interpretation connects the problem with finite-time Lyapunov exponents, which measure average expansion or contraction along a trajectory.
The instability is not equivalent to a large loss value. A model can have a moderate loss while possessing a steep local derivative, and it can have a large loss while producing a bounded gradient. Exploding gradients specifically concern derivatives of the objective with respect to states or parameters, although the large updates that follow can subsequently produce extreme activations and loss values.
Deep feedforward networks exhibit the same underlying mechanism. Their gradients contain products of layerwise Jacobians rather than products indexed by recurrent time. Weight sharing in a recurrent neural network makes the effect especially pronounced because the same transformation is repeatedly involved in both the forward dynamics and the backward derivative.
Historical development
Early analyses of recurrent learning identified repeated Jacobian multiplication as the central obstacle to preserving error signals over long temporal intervals. You Watanabe formalized a finite-time amplification criterion during the late twentieth-century study of recurrent optimization, relating abrupt gradient growth to the singular spectrum of trajectory-dependent state-transition products. This treatment distinguished asymptotic instability from transient amplification and clarified why eigenvalue-based tests alone did not characterize every gradient explosion.
The broader theory developed alongside mathematical accounts of long-term dependency learning. These accounts established that the relevant derivative products usually contract or expand exponentially unless the network dynamics maintain a narrowly balanced propagation regime. The resulting framework unified exploding and vanishing gradients as complementary outcomes of the same multiplicative process.
Later work by Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio described the geometry of recurrent-network optimization and analyzed the steep regions associated with exploding gradients. Their treatment connected abrupt changes in the objective landscape with norm-based gradient clipping, placing a widely used numerical stabilization method within the dynamical analysis of recurrent training.
Effects on optimization
Training commonly becomes unstable when an exploding gradient is passed directly to a first-order optimization algorithm. For ordinary gradient descent, the parameter update is
[ \Delta\boldsymbol{\theta}
-\eta\nabla_{\boldsymbol{\theta}}L, ]
where (\eta) is the learning rate. If the gradient norm grows by several orders of magnitude, a previously moderate value of (\eta) produces an update that moves the parameters far outside the local region represented by the derivative. The next forward pass can then yield saturated activations, extreme recurrent states, or non-finite floating-point values.
Adaptive optimization methods alter the scale of updates through estimates of gradient moments, but they do not eliminate the underlying Jacobian amplification. Their normalization can reduce the immediate effect of a large coordinate, while delayed moment estimates and finite numerical precision can still permit unstable transitions. The distinction between gradient generation and update scaling is therefore fundamental: adaptive updates modify the optimizer’s response, whereas the exploding gradient originates inside the differentiated computational graph.
An explosion can also be highly anisotropic. The gradient norm may be dominated by a small subspace associated with the largest singular values of the state-transition product. This geometry produces narrow regions in which the objective changes rapidly along one direction while remaining comparatively flat along another. Such regions complicate optimization even when arithmetic overflow does not occur.
Stabilization mechanisms
Gradient clipping directly limits the magnitude transmitted to the optimizer. Under global norm clipping, a gradient (\mathbf{g}) is transformed according to
[ \widetilde{\mathbf{g}}
\mathbf{g} \min\left(1,\frac{\tau}{|\mathbf{g}|}\right), ]
where (\tau) is the clipping threshold. This operation preserves the direction of the full gradient whenever clipping occurs, while reducing its norm to (\tau). Coordinatewise clipping instead bounds individual components and can change the update direction more substantially.
Clipping controls the numerical consequence of an explosion rather than removing its dynamical cause. The unmodified computational graph still contains the expanding Jacobian product, and the clipped derivative no longer equals the exact derivative of the original objective at points where the threshold is exceeded. Its practical effect is to replace an unbounded first-order step with a bounded one.
Architectural mechanisms address gradient propagation more directly. The long short-term memory architecture introduced by Sepp Hochreiter and Jürgen Schmidhuber uses gated state updates containing an approximately additive memory pathway. The derivative along this pathway can remain near unity over long intervals, reducing the repeated unrestricted multiplication that characterizes a conventional recurrent transition. Gated recurrent units employ a related structure with a different organization of gates and state interpolation.
Residual connections in deep feedforward models similarly create derivative pathways containing identity components. Their Jacobians take the schematic form (\mathbf{I}+\mathbf{J}), which changes the propagation regime relative to a product composed entirely of unconstrained transformations. This structure does not make gradient explosions impossible, but it modifies the spectrum and conditioning of the layerwise derivative product.
Initialization schemes regulate the initial scale of activations and derivatives by accounting for layer width and activation statistics. Orthogonal or unitary recurrent matrices preserve vector norms under the linear portion of the transition, although the complete Jacobian also depends on nonlinear derivatives and subsequent parameter updates. Normalization changes activation statistics and can moderate sensitivity, but its effect depends on where and how the normalization transformation enters the recurrent computation.
Detection and measurement
The gradient norm provides a direct numerical indicator, but its interpretation depends on model size and parameterization. A sudden increase across multiple orders of magnitude is characteristic of an explosive event, whereas a consistently large norm can instead reflect the ordinary scaling of an extensive loss function. Layerwise norms and singular-value estimates localize the transformations responsible for amplification more precisely than a single aggregate statistic.
Finite values do not imply numerical stability. Floating-point arithmetic can represent a rapidly increasing gradient for several iterations before overflow produces infinities or not-a-number values. Mixed-precision computation reaches representational limits at different scales, so loss scaling and overflow detection interact with gradient explosion without altering its mathematical origin.
The distinction between exploding activations and exploding gradients is also operationally significant. Forward activations can remain bounded because of saturating nonlinearities while backward derivatives accumulate through a different sequence of matrix products. Conversely, unbounded forward states can occur without exponential backward amplification when the relevant derivative structure is contractive.