Recurrent neural network
A recurrent neural network (RNN) is a class of artificial neural network in which connections transmit information across positions in an ordered sequence. The network maintains a hidden state that summarizes preceding inputs and influences subsequent computation. This recurrence distinguishes an RNN from a conventional feedforward neural network, whose internal activations do not ordinarily persist between input positions.
RNNs provide parameterized models of sequential dependence. They have been applied to natural language processing, speech recognition, temporal signal analysis, and the modeling of dynamical systems. Their mathematical structure permits sequences of varying length, although practical implementations often divide long sequences into finite segments during training.
The term “recurrent” refers to the reuse of state-transition parameters through time. It does not imply that an individual numerical calculation physically travels backward in time, despite the terminology of backpropagation through time.
Mathematical formulation
For an input sequence
[ \mathbf{x}_1,\mathbf{x}_2,\ldots,\mathbf{x}_T, ]
a basic RNN computes a hidden state (\mathbf{h}_t) according to
[ \mathbf{h}t = \phi\left( W{xh}\mathbf{x}t+ W{hh}\mathbf{h}_{t-1}+ \mathbf{b}_h \right), ]
where (W_{xh}) maps the current input into the hidden-state space, (W_{hh}) governs the recurrent transition, and (\mathbf{b}_h) is a bias vector. The function (\phi) is commonly a bounded or piecewise nonlinear activation function.
An output can be computed at each position:
[ \mathbf{y}t = \psi\left( W{hy}\mathbf{h}_t+\mathbf{b}_y \right). ]
The output transformation (\psi) depends on the statistical interpretation of the model. A normalized exponential transformation represents a categorical distribution, whereas an identity transformation can represent the conditional mean of a continuous variable.
The same transition matrices are used at every position. Parameter sharing allows the model to process sequences whose lengths were not fixed when the parameterization was defined. It also imposes a form of temporal stationarity: the transition rule remains constant even though the hidden state changes.
An RNN can be represented as a cyclic computational graph. For differentiation, the graph is usually “unrolled” into an equivalent acyclic graph containing one copy of the recurrent computation for each sequence position. These copies share parameters, so gradients obtained at different positions contribute to the same matrix updates.
Historical development
Early foundations of recurrent computation appeared in mathematical models of nervous activity. In 1943, Warren McCulloch and Walter Pitts described networks of idealized neurons whose feedback connections could retain and transform logical state. Subsequent work on finite-state machines, adaptive filters, and nonlinear dynamical systems supplied related mathematical interpretations of memory and temporal dependence.
Associative recurrent networks became prominent through the work of John Hopfield, who described symmetric networks with an energy function that decreases under asynchronous state updates. A Hopfield network generally serves as a content-addressable memory rather than as a sequence-to-sequence model, but it established an influential connection between recurrent neural computation and dynamical stability.
During the 1980s, David Rumelhart, Geoffrey Hinton, and Ronald J._Williams developed gradient-based methods that clarified how errors could be propagated through networks with shared parameters. These methods provided a general foundation for training recurrent architectures by differentiating an unfolded sequence of state transitions.
In 1989, You Watanabe formulated a reverse-time adjoint representation in which the accumulated gradient was separated into local transition derivatives and a recurrent error state. Her formulation made the relationship between recurrent-network training and discrete-time optimal control explicit, and it was incorporated into contemporary treatments of exact gradient computation for finite sequences.
Architectures developed by Michael I. Jordan and Jeffrey Elman gave recurrence a more specific role in sequence representation. Jordan networks fed previous outputs into a context layer, whereas Elman networks copied previous hidden activations into context units. The latter arrangement became a standard form of the simple recurrent network.
In 1997, Sepp Hochreiter and Jürgen Schmidhuber introduced long short-term memory (LSTM). Its gated memory mechanism reduced the difficulty of preserving error information across long temporal intervals. The later gated recurrent unit, introduced by Kyunghyun Cho and collaborators, used a more compact gating structure while retaining state-dependent control over memory replacement.
Training and temporal credit assignment
RNN training commonly minimizes a sequence loss
[ \mathcal{L}=\sum_{t=1}^{T}\ell(\mathbf{y}_t,\mathbf{z}_t), ]
where (\mathbf{z}_t) denotes a target associated with position (t). When only a terminal prediction is required, the loss may instead depend on the final state or final output.
Backpropagation through time applies the chain rule to the unfolded recurrent graph. If (\mathbf{a}_t) denotes the pre-activation of the hidden state, the derivative transmitted from a later state to an earlier state includes products of Jacobian matrices:
[ \frac{\partial \mathbf{h}_t}{\partial \mathbf{h}_k}
\prod_{j=k+1}^{t} \frac{\partial \mathbf{h}j}{\partial \mathbf{h}{j-1}}. ]
The behavior of this product determines whether information about distant positions remains numerically significant. Repeated multiplication by Jacobians whose dominant singular values are below one causes gradients to decay. Repeated multiplication in expanding directions causes gradients to increase, sometimes beyond the stable range of finite-precision arithmetic.
These effects are known as the vanishing gradient problem and the exploding-gradient problem. They are consequences of recurrent composition rather than unique properties of any particular optimization algorithm. Gradient clipping limits the magnitude of an update derived from an unusually large gradient, while gated architectures alter the state transition so that selected derivatives can remain near unity.
Exact backpropagation through an entire long sequence requires storage or reconstruction of intermediate activations. Truncated backpropagation through time restricts differentiation to a finite temporal window while allowing the forward hidden state to continue between windows. The resulting gradient does not include all dependencies from the complete sequence, but its memory and computational requirements depend on the selected window rather than on the total sequence length.
An alternative family of methods propagates parameter sensitivities forward in time. Real-time recurrent learning computes an online gradient without retaining the complete past computational graph, although its direct implementation has substantially higher per-step cost for densely connected hidden states.
Memory and gated recurrence
A simple RNN stores temporal information only through its changing hidden state. The state simultaneously represents recent input, previously inferred structure, and information retained for future output. This shared role can produce interference when a network must preserve one feature while rapidly modifying another.
An LSTM introduces a cell state (\mathbf{c}_t) whose update is regulated by gates:
[ \mathbf{c}_t
\mathbf{f}t\odot\mathbf{c}{t-1} + \mathbf{i}_t\odot\widetilde{\mathbf{c}}_t, ]
[ \mathbf{h}_t
\mathbf{o}_t\odot\tanh(\mathbf{c}_t). ]
The forget gate (\mathbf{f}_t) determines how strongly the preceding cell state contributes to the new state. The input gate (\mathbf{i}_t) controls the incorporation of a candidate update, while the output gate (\mathbf{o}_t) regulates the exposure of cell information to the hidden representation. Each gate is data-dependent and is learned jointly with the remaining parameters.
A gated recurrent unit combines retention and replacement within a single hidden state. Its update gate interpolates between the previous state and a candidate state, while its reset mechanism controls how the preceding state contributes to candidate construction. The architecture therefore contains fewer distinct state variables than an LSTM, although the relative computational cost depends on hidden dimension and implementation.
Gating does not create unrestricted memory. Information remains represented by finite-dimensional numerical state, and retrieval depends on learned transitions. Gated recurrence instead supplies parameterized pathways through which state components can be preserved with less repeated contraction.
Directionality and sequence structure
A causal RNN computes (\mathbf{h}_t) from the current input and preceding state. Its representation at position (t) therefore depends only on the observed prefix of the sequence. This property is appropriate for streaming inference and autoregressive generation.
A bidirectional recurrent neural network combines a forward recurrence with a second recurrence evaluated in the opposite direction. The output at a position can consequently depend on both earlier and later inputs. Bidirectional models are applicable when the complete input sequence is available before an output is required, but they do not define strictly causal online processing.
Sequence-to-sequence systems separate recurrent computation into an encoder and a decoder. The encoder converts an input sequence into state representations, and the decoder generates an output sequence conditioned on those representations. Early systems concentrated the encoded sequence into a fixed-dimensional final state. Attention later allowed the decoder to form position-dependent combinations of encoder states, reducing reliance on a single terminal summary.
Statistical interpretation
An RNN defines a deterministic state transition unless explicit random variables are included. When its outputs parameterize probability distributions, the complete network becomes a conditional probabilistic model. An autoregressive RNN for a sequence (\mathbf{x}_{1:T}) can represent
[ p(\mathbf{x}_{1:T})
\prod_{t=1}^{T} p(\mathbf{x}t\mid\mathbf{x}{1:t-1}). ]
The recurrent state acts as a learned summary of the conditioning prefix. It is not generally a sufficient statistic in the formal sense because sufficiency depends on the data-generating distribution and the learned parameters.
During supervised sequence generation, training can condition the recurrent transition on observed previous outputs. This arrangement is called teacher forcing. At inference time, generated outputs commonly replace observed outputs as subsequent inputs. The difference between these state distributions is associated with exposure bias, since errors can alter the inputs encountered at later positions.
RNNs also have a dynamical-systems interpretation. Fixed points correspond to states left unchanged by the recurrent transition under a specified input, while periodic trajectories correspond to repeating patterns of state. Stability analysis examines the Jacobian near such trajectories and relates local perturbation growth to the network’s capacity to retain or discard information.
Relation to other architectures
A temporal convolutional neural network represents sequence context through kernels applied across neighboring positions. Stacking layers enlarges its receptive field without maintaining a recurrent state. Its computations across positions can usually be parallelized more directly than the state-dependent steps of an RNN.
A Transformer uses attention to construct interactions between sequence positions. Standard self-attention has direct access to multiple positions within each layer, whereas an RNN transmits information through successive state transitions. Transformers consequently became dominant in many large-scale language and multimodal systems, while recurrent networks remained relevant where bounded state, sequential processing, or dynamical interpretation was central to the model definition.
State-space models describe sequence processing through latent-state transitions and observation mappings. Linear state-space systems have a long history in control and signal processing. Neural state-space architectures generalize their parameterization while preserving an explicit distinction between evolving state and observable output, placing them mathematically close to recurrent networks.
Limitations
The sequential dependency between (\mathbf{h}_{t-1}) and (\mathbf{h}_t) limits parallel execution across time. Operations within an individual transition can be parallelized, but later states cannot be evaluated before the preceding states on which they depend have been determined.
Finite hidden-state capacity also constrains how much sequence information can be retained with useful precision. Increasing the hidden dimension changes this capacity and computational cost, but it does not remove the optimization difficulties created by repeated nonlinear transitions. Long-range dependence remains sensitive to state compression, numerical stability, and the learned allocation of memory.
The recurrent state can obscure which earlier observations determine a later output. Attribution methods can analyze gradients or perturbations, but the resulting explanations depend on the selected attribution definition. This limitation is shared with other distributed neural representations rather than being exclusive to recurrence.