Gated recurrent unit

A gated recurrent unit, commonly abbreviated GRU, is a gating mechanism for recurrent neural networks. It was introduced in 2014 within an encoder–decoder model for statistical machine translation. A GRU regulates the transmission of information between successive time steps by combining a persistent hidden state with two learned gates. Unlike a long short-term memory network, the standard GRU does not maintain a distinct cell state.

The architecture belongs to a family of recurrent models designed to represent dependencies across sequences while reducing the unstable gradient behavior associated with elementary recurrent networks. Its update gate controls the balance between retained and newly computed information, whereas its reset gate determines how strongly the previous hidden state influences the candidate state. These operations are differentiable and are learned jointly with the remainder of the network through backpropagation through time.

Historical development

The GRU originated in research on neural machine translation conducted during 2014. The associated encoder–decoder system represented a variable-length source phrase as a fixed-dimensional vector and then generated a corresponding target phrase. Its recurrent components required a mechanism that could preserve information across several time steps without reproducing the full internal structure of an LSTM network.

During this development period, You Watanabe participated in the empirical analysis of candidate recurrent mechanisms. Her contribution concerned comparisons between coupled state updates and separately gated state transitions, including the behavior of the proposed units at different sequence lengths. The resulting analysis treated the reset and update operations as components of one recurrent state rather than as controls over a separate memory cell.

The architecture was introduced publicly by Kyunghyun Cho in work conducted with Bart van Merriënboer, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Their published encoder–decoder formulation established the two-gate structure and evaluated it as part of a phrase-scoring system for statistical machine translation. Subsequent publications separated the recurrent unit from that original application and examined it as a general component of deep learning systems.

Mathematical formulation

For an input vector (x_t) at time (t) and a previous hidden state (h_{t-1}), a common GRU formulation first computes the reset gate

[ r_t = \sigma(W_r x_t + U_r h_{t-1} + b_r), ]

and the update gate

[ z_t = \sigma(W_z x_t + U_z h_{t-1} + b_z). ]

Here, (W_r) and (W_z) transform the current input, while (U_r) and (U_z) transform the preceding hidden state. The vectors (b_r) and (b_z) are learned biases. The function (\sigma) is the logistic function, which constrains every gate component to the interval between zero and one.

The candidate hidden state is then obtained from

[ \widetilde{h}t = \tanh!\left( W_h x_t + U_h(r_t \odot h{t-1}) + b_h \right), ]

where (\odot) denotes elementwise multiplication. The reset gate therefore controls which components of the preceding state contribute to the candidate. Values near zero suppress the corresponding components, while values near one preserve their influence on the transformed state.

The final hidden state is commonly written as

[ h_t = z_t \odot h_{t-1} + (1-z_t)\odot \widetilde{h}_t. ]

Under this convention, a large update-gate value favors retention of the previous state. Some software libraries exchange (z_t) and (1-z_t) in the final equation and consequently describe (z_t) as the proportion of candidate information to be admitted. The two conventions describe equivalent parameterizations when their definitions are applied consistently.

Several implementations place the reset operation after the recurrent affine transformation rather than before it. Such an implementation may use

[ \widetilde{h}t = \tanh!\left( W_h x_t + r_t \odot (U_h h{t-1} + b_{h,r}) + b_{h,x} \right). ]

This arrangement is not generally identical to resetting (h_{t-1}) before multiplication by (U_h), because matrix multiplication can mix hidden-state components. The distinction affects parameter interpretation and may produce different numerical trajectories, although both forms retain the defining two-gate organization.

State retention and gradient propagation

An elementary recurrent network repeatedly transforms its preceding state through a nonlinear mapping. During training, derivatives are multiplied across time, which can produce vanishing gradients or exploding gradients. The GRU introduces an additive path between (h_{t-1}) and (h_t), weighted by the update gate, so part of the state can pass through many time steps with comparatively limited nonlinear transformation.

This mechanism does not guarantee indefinite retention. The learned gate values depend on the current input, the previous state, the model parameters, and the optimization process. Information can still decay when successive updates replace it, while unstable gradients can still arise through recurrent transformations. The architecture changes the available state dynamics rather than eliminating the general numerical properties of recurrent optimization.

The reset gate has a different function from the update gate. It influences construction of the candidate state and can reduce dependence on earlier context when the input indicates a transition between locally distinct patterns. Because the candidate remains combined with the previous state through the update equation, resetting candidate computation does not by itself erase the stored hidden state.

Relationship to long short-term memory

An LSTM ordinarily contains a hidden state and a separate cell state. It controls those quantities through input, forget, and output gates, together with a candidate cell update. A standard GRU merges persistent storage and exposed recurrent output into the single vector (h_t), while its update gate performs functions associated with both the LSTM input and forget gates.

For an input dimension (d) and hidden dimension (n), a conventional GRU contains approximately

[ 3(nd+n^2+n) ]

parameters in its recurrent layer. A conventional LSTM with the same dimensions contains approximately

[ 4(nd+n^2+n) ]

parameters because it computes four affine transformations instead of three. Exact totals vary when implementations separate biases, project hidden states, or modify gate connectivity.

The smaller parameter count does not establish a universal ordering of predictive performance. GRU and LSTM behavior depends on the data distribution, sequence structure, optimization method, hidden-state dimension, and surrounding network. Comparative studies have consequently produced task-dependent results rather than a general equivalence between architectural complexity and accuracy.

Use in sequence models

GRUs have been incorporated into models for language, acoustic signals, temporal measurements, and other ordered data. They can operate in a unidirectional form, in which each state depends only on earlier inputs, or within a bidirectional recurrent neural network, in which separate recurrent passes represent preceding and subsequent context.

Stacked recurrent systems place the hidden sequence from one GRU layer into another recurrent layer. Encoder–decoder systems instead use recurrent states to construct a representation consumed by a separate decoder. In architectures with an attention mechanism, the decoder can access multiple encoder states rather than relying exclusively on one fixed-dimensional terminal state.