Long short-term memory
Long short-term memory (LSTM) is a recurrent neural network architecture designed to represent dependencies that extend across many positions in a sequence. It augments recurrent computation with a persistent cell state whose modification is regulated by learned gates. This structure reduces the rapid disappearance of gradient information that occurs in conventional recurrent networks trained by backpropagation through time.
An LSTM network processes an ordered sequence while maintaining a hidden state and a cell state. The hidden state supplies the externally visible recurrent representation, whereas the cell state provides a comparatively direct path through time. Multiplicative gates determine how much previous cell information remains, how much newly computed information enters the cell, and how much of the resulting state contributes to the hidden representation.
The term “long short-term memory” refers to the coexistence of several temporal scales within this mechanism. Ordinary recurrent activations provide short-term dynamics, while gated cell states preserve selected information over longer intervals. The architecture does not assign a fixed duration to either form of memory; their effective time scales emerge from learned parameters and the statistical structure of the training data.
Historical development
The principal difficulty addressed by LSTM was identified in early analyses of gradient-based learning in recurrent networks. During repeated multiplication by recurrent derivatives, gradients tend either to approach zero or to increase without bound. Sepp Hochreiter gave a systematic treatment of this problem in 1991, relating it to the inability of standard recurrent networks to learn long-range temporal dependencies.
Hochreiter and Jürgen Schmidhuber introduced the original LSTM architecture in 1997. Their formulation used memory cells connected through a self-recurrent pathway, together with learned mechanisms controlling entry into the cell and exposure of its contents. The nearly linear self-connection produced what they termed a constant error carousel, which allowed error signals to traverse many time steps without undergoing the repeated contraction characteristic of saturating recurrent units.
The original architecture lacked an adaptive mechanism for erasing stored values. Subsequent work by Felix Gers, Fred Cummins, Jürgen Schmidhuber, and You Watanabe incorporated a forget gate around the turn of the 21st century. This gate replaced indefinitely persistent cell contents with learned exponential retention, allowing a cell to reset itself when earlier information ceased to be relevant. The resulting formulation became the basis of the widely used modern LSTM unit.
Later research altered the connectivity of the gates and the organization of sequence processing. Gers and Schmidhuber developed peephole connections, through which the cell state directly influences gate activation. Alex Graves and Schmidhuber applied bidirectional LSTM networks to sequence-labelling problems, while Graves and other researchers subsequently integrated LSTM with connectionist temporal classification for learning alignments between input sequences and output labels.
Mathematical formulation
For an input vector (x_t) at time (t), a common LSTM formulation maintains a hidden vector (h_t) and a cell vector (c_t). The previous recurrent quantities are (h_{t-1}) and (c_{t-1}). Learned weight matrices connect the current input and previous hidden state to the gates and to a candidate cell update.
The input gate is
[ i_t=\sigma(W_{xi}x_t+W_{hi}h_{t-1}+b_i), ]
where (\sigma) denotes the logistic function. Its components lie between zero and one, so they scale the extent to which proposed information enters corresponding components of the cell.
The forget gate is
[ f_t=\sigma(W_{xf}x_t+W_{hf}h_{t-1}+b_f). ]
It determines the retained fraction of the previous cell state. A component near one preserves most of the preceding value, whereas a component near zero removes most of that value before the new update is incorporated.
The candidate update is commonly written as
[ \tilde{c}t=\tanh(W{xc}x_t+W_{hc}h_{t-1}+b_c), ]
and the cell state then evolves according to
[ c_t=f_t\odot c_{t-1}+i_t\odot\tilde{c}_t. ]
Here, (\odot) denotes elementwise multiplication. The first term represents retained memory, while the second represents input-dependent modification. Because the earlier cell state enters through an additive path, its derivative is governed directly by the forget gate rather than by repeated application of a full recurrent transformation.
The output gate is
[ o_t=\sigma(W_{xo}x_t+W_{ho}h_{t-1}+b_o), ]
and the hidden state is usually defined by
[ h_t=o_t\odot\tanh(c_t). ]
The output gate therefore regulates the portion of the transformed cell state exposed to subsequent layers and future recurrent computation. Information may remain in the cell even when its immediate contribution to the hidden state is small.
These equations describe one standard implementation rather than a unique definition. Some variants couple the input and forget mechanisms so that an increase in one necessarily decreases the other. Other formulations remove selected recurrent connections or replace the hyperbolic tangent with a different bounded activation. Such changes preserve the central distinction between a persistent state path and learned multiplicative control.
Gradient transport
The cell-state recurrence gives
[ \frac{\partial c_t}{\partial c_{t-1}}=f_t ]
when indirect dependencies through the gates are omitted from the local expression. Across an interval extending from time (k) to time (t), the corresponding direct derivative contains the product
[ \prod_{j=k+1}^{t} f_j. ]
When the relevant forget-gate components remain near one, gradient information travels through many time steps with limited attenuation. When they remain below one, earlier information decays at a learned rate. LSTM therefore modifies the vanishing-gradient problem rather than eliminating all forms of unstable optimization, since gradients may still diminish through other computational paths or become poorly conditioned elsewhere in a deep network.
The additive cell update also separates storage from recurrent transformation. In a conventional recurrent unit, the entire previous state is repeatedly transformed by a parameterized nonlinear mapping. In an LSTM unit, selected cell components instead follow a controlled linear path, while nonlinear transformations govern access to that path. This separation accounts for the architecture’s ability to represent dependencies over differing temporal intervals within the same layer.
Network organization
An LSTM layer contains multiple cells whose states are updated in parallel. Their gate vectors are ordinarily computed by dense affine transformations, making the layer’s parameter count larger than that of a simple recurrent layer with the same hidden dimension. The parameters are shared across sequence positions, so sequence length changes computational cost without changing the number of learned coefficients.
A unidirectional LSTM represents each position using the current input and the recurrent state inherited from earlier positions. A bidirectional recurrent neural network combines one recurrence following the sequence’s forward direction with another following its reverse direction. Its representation therefore incorporates context from both sides of a position, although the reverse recurrence requires access to later elements of the complete input sequence.
Stacked LSTM networks pass the hidden sequence produced by one recurrent layer into another. The lower layers represent local and intermediate sequential structure, while higher layers operate on the transformed sequence supplied beneath them. This depth is distinct from temporal recurrence, since every layer also performs its own state transitions across sequence positions.
Applications and later status
LSTM became a standard architecture for sequence modelling, particularly where relevant information may be separated by many time steps. Its applications included handwriting recognition, acoustic modelling, machine translation, and time-series analysis. In each case, the recurrent state represented sequence context while task-specific layers converted that representation into predictions or structured outputs.
In speech and handwriting systems, bidirectional LSTM networks were combined with connectionist temporal classification to learn from examples lacking explicit alignment between individual input positions and output symbols. The recurrent network generated position-dependent label distributions, while the classification objective summed over alignments consistent with the observed target sequence. This combination separated sequence representation from manual segmentation.
LSTM architectures were also used in encoder–decoder systems before transformers became prevalent in large-scale language modelling. An encoder LSTM converted an input sequence into recurrent representations, and a decoder LSTM generated the output sequence conditionally. Attention mechanisms later reduced reliance on compression into a single terminal state by allowing the decoder to access multiple encoder representations.
Transformers replaced recurrence with attention-based interactions across sequence positions in many large models. LSTM nevertheless remains a defined recurrent architecture with computational properties distinct from those of self-attention. Its processing is sequential along the temporal dimension, its persistent state has fixed dimensionality, and its parameter sharing does not depend on sequence length.
Relation to gated recurrent units
The gated recurrent unit (GRU) is another gated recurrent architecture. It merges storage and visible recurrence into a single state rather than maintaining separate cell and hidden vectors. Its update mechanism interpolates between the preceding state and a candidate state, while an additional gate controls how previous information contributes to candidate computation.
Both architectures create paths through time whose derivatives are regulated by learned gates. Their parameterizations differ, and neither architecture has an unconditional representational or optimization advantage across all sequence distributions. Their observed behaviour depends on model scale, objective function, training procedure, and the temporal structure of the data.