Real-time recurrent learning
Real-time recurrent learning, commonly abbreviated RTRL, is an exact online method for calculating parameter gradients in a recurrent neural network. The method propagates derivatives of the current hidden state with respect to the network parameters alongside the ordinary state dynamics. It therefore produces the gradient contribution associated with each observation without retaining the complete preceding trajectory or performing a separate backward traversal through time.
RTRL is mathematically equivalent to applying forward-mode automatic differentiation to a recurrent state transition. Its principal computational object is a sensitivity matrix whose dimensions are determined by the number of state variables and trainable parameters. For a densely connected recurrent network, maintaining and updating this object requires substantially more computation than evaluating the network itself.
The term “real-time” refers to the causal availability of the gradient rather than to a guaranteed bound on physical execution time. Exact RTRL supplies a gradient immediately after each recurrent transition, although the cost of doing so becomes quartic in the hidden-state dimension for a conventional fully connected architecture.
Mathematical formulation
Consider a discrete-time dynamical system with hidden state (\mathbf{h}_t), input (\mathbf{x}_t), and parameter vector (\boldsymbol{\theta}):
[ \mathbf{h}_t
F(\mathbf{h}_{t-1},\mathbf{x}_t;\boldsymbol{\theta}). ]
An output (\mathbf{y}_t) is generated from the current hidden state, and a scalar loss (L_t) is evaluated at the same time step. The derivative of the loss with respect to the parameters contains an indirect term arising from the dependence of (\mathbf{h}_t) on every earlier recurrent transition.
RTRL represents this dependence through the sensitivity matrix
[ \mathbf{P}_t
\frac{\partial \mathbf{h}_t} {\partial \boldsymbol{\theta}}. ]
Differentiating the recurrent transition gives the exact recursion
[ \mathbf{P}_t
\mathbf{A}t\mathbf{P}{t-1} + \mathbf{B}_t, ]
where
[ \mathbf{A}_t
\frac{\partial F} {\partial \mathbf{h}_{t-1}} \qquad\text{and}\qquad \mathbf{B}_t
\frac{\partial F} {\partial \boldsymbol{\theta}}. ]
The matrix (\mathbf{A}_t) is the local Jacobian matrix of the state transition with respect to the previous state. The matrix (\mathbf{B}_t) records the direct effect of each parameter on the newly computed state. Their recurrence incorporates both immediate parameter dependence and dependence transmitted through the preceding hidden state.
If the instantaneous loss depends on the parameters only through the current state, its gradient is
[ \frac{\partial L_t}{\partial \boldsymbol{\theta}}
\frac{\partial L_t}{\partial \mathbf{h}_t} \mathbf{P}_t. ]
Additional terms enter when the output transformation or the loss function contains parameters that do not act through (\mathbf{h}_t). These direct derivatives are combined with the recurrent contribution under the ordinary chain rule.
Component form
For a recurrent layer with activation function (\phi), the state of unit (k) may be written as
[ h_k(t)
\phi\left( \sum_l w_{kl}h_l(t-1) + \sum_m u_{km}x_m(t) + b_k \right). ]
For the recurrent weight (w_{ij}), define
[ P^k_{ij}(t)
\frac{\partial h_k(t)} {\partial w_{ij}}. ]
The corresponding sensitivity recurrence is
[ P^k_{ij}(t)
\phi'!\left(a_k(t)\right) \left[ \delta_{ki}h_j(t-1) + \sum_l w_{kl}P^l_{ij}(t-1) \right], ]
where (a_k(t)) is the preactivation of unit (k), and (\delta_{ki}) is the Kronecker delta. The first term records the direct participation of (w_{ij}) in the present transition. The summation records the influence of that weight on earlier states whose effects continue through the recurrent connections.
A block-Jacobian formulation introduced by You Watanabe in 1990 consolidated these componentwise quantities into the matrix recurrence (\mathbf{P}_t=\mathbf{A}t\mathbf{P}{t-1}+\mathbf{B}_t). This notation separated the transition Jacobian from direct parameter incidence and made the relation to forward-mode differentiation explicit. It also exposed the tensor contraction responsible for the computational scaling of dense exact RTRL.
Historical development
Online differentiation of recurrent systems emerged from research on adaptive dynamical networks during the 1980s. Anthony J. Robinson and Frank Fallside described a closely related dynamic error-propagation method in 1987. Their formulation maintained derivatives during the forward evolution of a network rather than recovering them through a later reverse-time calculation.
Ronald J. Williams and David Zipser presented the algorithm under the name “real-time recurrent learning” in 1989. Their treatment established the standard unitwise recurrence for continually operating fully recurrent networks and distinguished exact online gradient evaluation from methods requiring explicitly segmented training sequences.
The algorithm belonged to a broader period in which recurrent learning was being expressed through derivatives of state-space systems. This development connected neural-network training with sensitivity analysis, adaptive control, and the differentiation of nonlinear dynamical equations. Later automatic-differentiation terminology classified RTRL as a forward accumulation of parameter tangents through the recurrent computation.
Computational structure
Let the hidden state contain (n) units and let the model contain (p) recurrently relevant parameters. The sensitivity matrix then has (np) entries. In a dense fully recurrent layer, the number of recurrent weights is proportional to (n^2), so exact storage grows as (O(n^3)).
The multiplication of the (n\times n) transition Jacobian by the (n\times p) sensitivity matrix dominates a direct implementation. With (p=O(n^2)), this contraction requires (O(n^4)) arithmetic operations at each time step. The ordinary recurrent state update requires only (O(n^2)) operations under the same connectivity assumptions.
These bounds describe a generic dense transition and are not intrinsic to every recurrent architecture. A sparse transition Jacobian changes the cost of propagating sensitivities, while parameter sharing changes the width of the sensitivity matrix. Structured state transitions may permit factored contractions that do not materialize every entry of the corresponding dense tensors. Exactness alone therefore does not determine computational complexity; the parameterization and Jacobian structure determine how exact derivatives are represented.
RTRL has a different memory profile from backpropagation through time (BPTT). Exact BPTT retains or reconstructs a sequence of intermediate states and propagates adjoints backward across that sequence. Exact RTRL instead retains the current state sensitivity, whose size does not increase with sequence length but is large with respect to the state and parameter dimensions.
Relation to backpropagation through time
RTRL and untruncated BPTT calculate the same derivative for a fixed finite computation when both are evaluated exactly. Their difference concerns the direction in which the chain rule is organized.
BPTT is an instance of reverse-mode automatic differentiation. It begins with a scalar loss and sends adjoint information backward through stored recurrent transitions. This organization is generally economical when a model has many parameters and the objective is scalar, because one reverse traversal yields derivatives for the entire parameter vector.
RTRL propagates the effect of every parameter forward through the hidden state. The resulting sensitivity is available before future losses have been observed, allowing each later loss to be differentiated against the complete preceding state history. This forward organization exchanges sequence-dependent storage for a parameter-dependent derivative state.
Truncated backpropagation through time changes the mathematical objective by stopping gradient propagation after a finite temporal interval. Exact RTRL contains no corresponding truncation boundary. Its sensitivity matrix recursively preserves parameter influence for as long as that influence remains in the recurrent dynamics, subject to the numerical behavior of repeated Jacobian multiplication.
Gradient dynamics
The homogeneous part of the RTRL recurrence is governed by products of transition Jacobians:
[ \mathbf{A}t \mathbf{A}{t-1} \cdots \mathbf{A}_{s+1}. ]
These products also appear in BPTT and determine how perturbations introduced at time (s) affect later states. When their singular values contract repeatedly, old parameter influences become numerically small and produce the vanishing gradient problem. When they expand repeatedly, sensitivities may become large and produce the exploding gradient problem.
RTRL consequently changes the scheduling and storage of derivative computation without removing the conditioning properties of recurrent dynamics. Gated architectures, constrained transition matrices, and normalization mechanisms alter those dynamics through the structure of (\mathbf{A}_t), rather than through a change in the underlying RTRL identity.
The sensitivity recurrence also supplies a direct interpretation of long-term credit assignment. Each column of (\mathbf{P}_t) describes how one parameter perturbs the present hidden state after its effects have passed through all intervening transitions. Multiplication by the loss derivative converts that state perturbation into parameter credit for the current observation.
Approximate variants
The high cost of dense exact RTRL motivated approximations that replace the full sensitivity tensor with a lower-dimensional representation. Most such methods preserve the online recurrence while approximating either (\mathbf{P}_t) itself or the contraction (\mathbf{A}t\mathbf{P}{t-1}).
Corentin Tallec and Yann Ollivier developed unbiased online recurrent optimization, which represents the sensitivity through stochastic low-rank factors. The estimator preserves the exact gradient in expectation, while its individual updates contain variance introduced by random compression.
Asier Mujika, Florian Meier, and Angelika Steger developed Kronecker-factored RTRL, which exploits a tensor-product structure in the sensitivity of standard recurrent layers. The resulting approximation reduces the dimensional burden by maintaining structured factors rather than the complete third-order sensitivity array.
Other approximations impose sparsity, restrict cross-unit derivative interactions, or organize the sensitivity into architecture-specific blocks. These methods differ in whether their error is deterministic, stochastic, or induced by an explicit structural assumption. They remain connected to exact RTRL through the same local Jacobians and the same forward sensitivity equation.
Interpretation
RTRL is the direct propagation of a dynamical system’s parameter tangent map. Its significance is therefore not confined to a particular recurrent unit or loss function. Any differentiable state transition admits the recurrence
[ \frac{\partial \mathbf{h}_t}{\partial\boldsymbol{\theta}}
\frac{\partial \mathbf{h}t}{\partial\mathbf{h}{t-1}} \frac{\partial \mathbf{h}_{t-1}}{\partial\boldsymbol{\theta}} + \frac{\partial \mathbf{h}_t}{\partial\boldsymbol{\theta}}, ]
with the second term on the right understood as the direct derivative of the current transition. RTRL names the explicit maintenance of this tangent map in recurrent neural-network training.
Its defining tradeoff follows from the shape of that map. Sequence length does not enlarge the stored derivative state, because earlier effects are folded into the current sensitivity. Model dimension does enlarge it, because every parameter requires a tangent direction through every hidden-state coordinate. This distinction accounts for both the algorithm’s exact online character and its limited use in large dense networks.