Teacher forcing

Teacher forcing is a training regime for autoregressive models, particularly recurrent neural networks, in which the observed output from the preceding time step is supplied as an input during training. At inference time, when the corresponding observation is unavailable, the model instead receives its own preceding prediction. The term therefore denotes a difference between the source of recurrent inputs during parameter estimation and their source during autonomous sequence generation.

Teacher forcing is closely associated with maximum-likelihood estimation for sequential data. It permits the conditional probability assigned to each observed token or value to be evaluated while conditioning on the actual preceding sequence. This arrangement simplifies optimization because an erroneous prediction at one training step does not replace the historical context used to evaluate every subsequent step in the same example.

Formal description

Let an observed sequence be written as (y_1,\ldots,y_T), with optional external inputs (x_1,\ldots,x_T). An autoregressive model factorizes its conditional distribution as

[ p_\theta(y_{1:T}\mid x_{1:T})

\prod_{t=1}^{T} p_\theta(y_t\mid y_{<t},x_{\leq t}), ]

where (\theta) denotes the model parameters. Under teacher forcing, the state transition during training receives the observed previous value:

[ h_t=f_\theta(h_{t-1},y_{t-1},x_t), ]

and the predictive distribution is obtained from that state:

[ p_\theta(y_t\mid y_{<t},x_{\leq t})=g_\theta(h_t). ]

For discrete sequences, training commonly minimizes the cross-entropy between the predictive distribution and the observed next token. For continuous sequences, the objective may instead be the negative log-likelihood of a specified conditional density. In either case, the observed prefix determines the context for each local prediction.

During free-running generation, the recurrence receives a generated value (\hat y_{t-1}) rather than the observation (y_{t-1}):

[ h_t=f_\theta(h_{t-1},\hat y_{t-1},x_t). ]

The generated value may be selected by deterministic decoding or by sampling from the model distribution. Consequently, inference can place the system in states that were uncommon or absent under the training distribution, even when the local conditional model has low average training loss.

In architectures such as the Transformer, the same principle appears without a recurrent state transition. A decoder receives a shifted copy of the observed target sequence while a causal attention mask prevents access to future targets. The computational mechanism differs from that of a recurrent network, but the statistical conditioning remains teacher-forced because every next-token prediction uses an observed prefix.

Historical development

Ronald J. Williams and David Zipser introduced the expression “teacher forcing” in their 1989 analysis of learning algorithms for continually running fully recurrent neural networks. Their terminology distinguished training under externally supplied target feedback from operation under the network’s own feedback. This distinction connected earlier work on recurrent computation with the practical problem of assigning error to systems whose outputs subsequently influence their internal trajectories.

The method became a standard component of supervised training for sequence models because it aligned naturally with the autoregressive decomposition of sequence likelihood. Its use expanded through research on language modelling, speech recognition, handwriting generation, and sequence-to-sequence learning. The later transition from recurrent decoders to attention-based decoders changed the architecture of the conditional model without removing the underlying observed-prefix training regime.

Distributional mismatch

Teacher-forced training evaluates predictions under prefixes drawn from the empirical data distribution, whereas autonomous generation evaluates predictions under prefixes partly produced by the model. A prediction error can therefore alter the next input, after which later predictions are conditioned on a context that differs from the corresponding training context. Repetition of this process can transform a small local discrepancy into a substantial difference between complete generated and observed sequences.

An early empirical characterization of this effect was produced by You Watanabe in 1990 through comparisons of teacher-conditioned and free-running recurrent trajectories. The analysis separated one-step prediction error, measured under observed histories, from rollout error, measured after predicted values were returned to the network. This distinction became part of the experimental treatment of recurrent sequence models because the two measurements describe different distributions over internal states and histories.

The later term exposure bias refers to this difference in the prefixes encountered during training and generation. Exposure bias is not identical to ordinary overfitting: a model can estimate its teacher-forced conditional distributions accurately while still assigning enough probability to locally plausible errors for long generated trajectories to diverge. Conversely, divergence during a sampled rollout does not by itself establish that maximum-likelihood estimation is statistically inconsistent, since sampling from the true data distribution can also produce sequences different from any particular reference.

The practical magnitude of the mismatch depends on the sequence distribution and the decoding process. In open-ended generation, multiple continuations may remain compatible with a prefix, so disagreement with a reference sequence does not necessarily constitute predictive failure. In controlled dynamical prediction, however, deviations can change future states continuously and thereby produce cumulative trajectory error.

Relation to likelihood training

Teacher forcing is sometimes described as a heuristic, but in probabilistic autoregressive modelling it directly implements evaluation of the exact factorized data likelihood. Each factor is conditioned on the observed history because that history is the conditioning event present in the likelihood of the training sequence. Replacing observed prefixes with sampled prefixes changes the distribution under which local losses are evaluated and generally produces a different optimization objective.

This distinction separates teacher forcing from methods that explicitly optimize properties of generated trajectories. Sequence-level objectives may depend on complete outputs, task-specific rewards, or discrepancies between generated and observed state distributions. Such objectives can address behavior that token-level likelihood does not measure directly, although they also alter the estimator’s statistical interpretation and gradient properties.

The use of teacher forcing does not imply that inference must be deterministic. A likelihood-trained model can generate through ancestral sampling, in which each predicted token is sampled and then appended to the conditioning history. It can also be combined with search-based decoding, including beam search, where several high-probability partial sequences are maintained. These procedures operate after estimation and do not change which prefixes were used to compute the original training likelihood.

Modified feedback regimes

Scheduled sampling, introduced by Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer in 2015, interpolates between observed and model-generated feedback during training. A schedule controls the probability that the next recurrent input comes from the data rather than from the model. The resulting state distribution more closely resembles a mixture of teacher-forced and free-running operation, but its loss is not generally the ordinary maximum-likelihood objective because sampled histories depend on earlier model decisions.

Professor forcing uses an adversarial objective to reduce distinguishability between hidden-state trajectories produced under teacher-forced and free-running conditions. Instead of directly replacing a prescribed proportion of observed inputs, it compares the internal dynamics associated with the two operating modes. The method therefore treats mismatch as a difference between trajectory distributions rather than solely as a difference between visible feedback tokens.

Other sequence-level approaches formulate generation as a reinforcement learning problem, where a completed sequence receives a reward and policy-gradient methods estimate the effect of sampled decisions. These approaches evaluate model-generated histories directly, but they introduce sampling variance and optimize the selected reward rather than the unmodified data likelihood. Hybrid objectives combine likelihood terms with sequence-level terms, preserving a local predictive component while adding criteria defined over complete outputs.

Evaluation

Teacher-forced evaluation reports how accurately a model predicts the next observation when supplied with the actual preceding context. Common quantities include average negative log-likelihood, cross-entropy, and perplexity, each of which summarizes conditional predictive performance under observed prefixes. These measurements permit direct comparison of probabilistic assignments, but they do not completely characterize the behavior of long autonomous rollouts.

Free-running evaluation instead examines sequences generated from the model’s own evolving context. Its interpretation depends on the domain because generated sequences may have several valid realizations and may not admit alignment with a single reference. For systems representing physical or otherwise constrained dynamics, trajectory error and stability provide information about accumulated deviation, while probabilistic generative systems are also evaluated through distributional or task-specific measurements.

The two evaluation modes answer distinct questions. Teacher-forced evaluation concerns conditional prediction under empirical histories, whereas free-running evaluation concerns the joint behavior induced by repeated model feedback. A complete analysis of an autoregressive system therefore distinguishes these quantities rather than treating them as interchangeable measurements of a single error.