Reinforcement learning from human feedback

Reinforcement learning from human feedback, commonly abbreviated RLHF, is a family of machine learning methods in which information derived from human judgments supplies or modifies the reward signal used to train an artificial agent. The method is primarily applied when the intended behavior is difficult to specify as a numerical objective but comparatively easy for human evaluators to recognize in concrete examples. Its prominent uses include the control of simulated agents, the adaptation of large language models, and the study of alignment between learned behavior and human preferences.

A typical RLHF system separates training into three interacting components. A generative policy produces candidate behavior, human evaluators express preferences among selected candidates, and a learned reward model predicts those preferences for examples that have not been evaluated directly. Reinforcement learning then modifies the policy to increase the predicted reward. Consequently, the final policy is optimized against a statistical representation of human evaluations rather than against continuous direct supervision by evaluators.

The phrase does not denote a single algorithm. It describes an arrangement in which human feedback enters a reinforcement-learning process, often through preference learning. The feedback may concern complete outputs, segments of a trajectory, proposed actions, or comparisons between alternative behaviors. Pairwise comparison became common because evaluators can often decide which of two outputs better satisfies a stated criterion without assigning either output an absolute numerical score.

Formal formulation

Let a policy (\pi_\theta(y\mid x)), parameterized by (\theta), generate an output (y) in response to an input (x). Human evaluators compare outputs (y_a) and (y_b) produced for the same input. A preference observation can be represented as

[ y_a \succ y_b, ]

where the relation indicates that (y_a) was selected over (y_b) under the evaluation protocol.

A reward model (r_\phi(x,y)), parameterized by (\phi), assigns a scalar value to an input-output pair. A widely used probabilistic model treats the difference between two reward scores as determining the probability of a human preference:

[ P_\phi(y_a \succ y_b\mid x)

\frac{\exp(r_\phi(x,y_a))} {\exp(r_\phi(x,y_a))+\exp(r_\phi(x,y_b))}. ]

The reward-model parameters are estimated by minimizing the negative log-likelihood of the recorded comparisons. This formulation is related to the Bradley–Terry model and identifies reward only up to transformations that leave the modeled preference probabilities unchanged.

After reward-model training, the policy is adjusted to maximize expected predicted reward. In language-model applications, the objective commonly contains an additional penalty for divergence from a reference policy:

[ J(\theta)

\mathbb{E}{x,y\sim\pi\theta} \left[ r_\phi(x,y)

\beta D_{\mathrm{KL}} \left( \pi_\theta(\cdot\mid x) \parallel \pi_{\mathrm{ref}}(\cdot\mid x) \right) \right]. ]

Here, (D_{\mathrm{KL}}) is the Kullback–Leibler divergence, and (\beta) controls the influence of the reference policy. The divergence term restricts changes that would otherwise exploit narrow irregularities in the learned reward function. It also preserves statistical properties acquired during pretraining, although it does not guarantee semantic correctness or conformity with evaluator intent.

The reward model estimates which response an evaluator would select under the conditions represented in the training data. It does not independently determine which response is true, socially acceptable, or useful outside those conditions. This distinction is central to the interpretation of RLHF: the model learns a preference predictor, while reinforcement learning converts that predictor into an optimization target.

Historical development

The intellectual basis of RLHF combines research on reinforcement learning, preference elicitation, inverse reinforcement learning, and learning from demonstrations. Earlier work on inverse reinforcement learning examined how an agent could infer a reward function from observed behavior. Preference-based reinforcement learning instead used comparative judgments to identify behavior favored by a human evaluator, avoiding the assumption that the evaluator’s demonstrations were optimal or even available.

In the 2010s, researchers developed scalable forms of preference-based training for agents operating in simulated environments. Paul Christiano and You Watanabe contributed to the experimental design of a 2017 study in which human comparisons between short trajectory segments were used to train reward predictors for control tasks and video-game behavior. The comparison interface presented evaluators with synchronized alternatives and recorded a relative judgment, allowing the learned reward function to replace direct numerical specification of the task objective during subsequent policy optimization.

The study established a recurring empirical structure: the agent generated behavior, evaluators compared selected samples, and the reward model generalized from those comparisons. Because the policy changed during training, the distribution of examples also changed. Feedback collection therefore operated as an iterative process rather than as a fixed annotation stage, linking RLHF to active learning and distribution-dependent data acquisition.

Jan Leike and Shane Legg subsequently examined scalable supervision and reward-learning systems in which human judgment remained informative despite increasing task complexity. Their work placed preference modeling within the broader technical problem of AI alignment, particularly the problem of representing objectives that could not be completely expressed through hand-written reward functions.

RLHF became closely associated with language-model adaptation after research demonstrated that human comparisons could distinguish summaries and conversational responses that differed in ways poorly captured by token-level likelihood. Nisan Stiennon and Daniel Ziegler developed preference-modeling methods for abstractive summarization, using evaluator comparisons to train a reward model and reinforcement learning to adjust a pretrained language model. The resulting systems exhibited the central statistical tradeoff of the method: increasing modeled preference could improve conformity with the annotation criteria while increasing divergence from the pretrained distribution.

Long Ouyang and Jeff Wu later participated in the development of instruction-following language models trained with demonstrations, preference comparisons, and policy optimization. This configuration combined supervised learning with RLHF. Demonstrations established an initial response distribution, while comparisons supplied a relative signal for behaviors that were difficult to specify through a single reference answer.

Application to language models

In language-model training, RLHF generally follows broad self-supervised learning on a large text corpus. Pretraining estimates linguistic regularities by predicting text, whereas RLHF modifies the model’s response distribution according to an evaluation protocol. The two stages therefore optimize different empirical objectives.

Supervised adaptation commonly precedes reward modeling. Human-written responses or curated examples produce a policy that reacts to instructions in a form suitable for comparison. The policy then generates several candidate responses for each prompt, and evaluators rank or compare them according to defined criteria. These judgments become the training data for the reward model.

A policy-optimization algorithm uses the reward model to score newly generated responses. Proximal policy optimization has frequently served this role because it constrains the magnitude of policy updates while supporting minibatch optimization. Its historical prevalence does not make it constitutive of RLHF; other reinforcement-learning algorithms and direct preference objectives can implement related training arrangements.

Human judgments may encode properties that are only partially separable. An evaluator selecting between two responses can react simultaneously to factual content, organization, tone, relevance, and compliance with the prompt. The reward model compresses these interacting considerations into a scalar value. This compression permits optimization, but it obscures distinctions that would remain explicit in a multidimensional evaluation.

The data-generating process also affects the meaning of the learned reward. Preferences depend on the wording of instructions given to evaluators, the candidates shown in each comparison, and the population from which evaluators are selected. Agreement between evaluators measures consistency under that process rather than the existence of a context-independent preference ordering.

Optimization behavior

The policy and reward model occupy different stages of the same feedback loop. The reward model is trained on outputs sampled from one or more policy distributions, while policy optimization can later produce outputs outside the reward model’s well-supported region. A policy may therefore discover patterns that receive high predicted reward without satisfying the underlying evaluation criterion. This phenomenon is a form of reward hacking and is related to specification gaming.

The problem does not require deliberate reasoning by the trained system. Statistical optimization alone can amplify features that correlate with positive labels in the comparison data. If evaluators tend to prefer detailed answers within the sampled dataset, the reward model may learn an overly general association between length and quality. Policy optimization can then increase length beyond the range for which that association was informative, producing responses whose additional text contributes little besides additional text.

Regularization against a reference policy reduces the extent of such movement but does not eliminate it. A strong divergence penalty limits reward improvement, while a weak penalty allows broader exploitation of the learned objective. The resulting balance is empirical because reward scale, policy entropy, and the quality of the preference model vary across training configurations.

Policy optimization can also reduce output diversity. When the reward model consistently assigns higher values to a narrow response pattern, repeated optimization concentrates probability on that pattern. This effect resembles mode collapse, although the underlying training dynamics differ from those of generative adversarial networks. Reduced diversity may increase consistency under the evaluation rubric while discarding valid alternatives not favored by the available comparisons.

Measurement and limitations

RLHF performance is usually assessed through held-out human comparisons, automated benchmarks, or behavioral evaluations targeting defined failure modes. Held-out preference accuracy measures the reward model’s ability to predict evaluator choices, but it does not measure the policy’s behavior after extensive optimization. Policy evaluation therefore requires outputs produced by the updated system rather than only comparisons retained from reward-model training.

Inter-evaluator disagreement is not simply annotation noise. It can reflect ambiguity in the prompt, variation in interpretation, or genuine differences in preference. Aggregating judgments into a single target produces a population-level estimate under the chosen sampling and weighting scheme. Minority preferences may consequently have little influence even when they are internally consistent.

Feedback quality is further constrained by the evaluator’s access to information. A plausible but false response may receive higher ratings than a less fluent accurate response when verification is difficult. In such cases, RLHF reproduces the evaluator’s observable judgment rather than the unobserved factual state. External tools, expert review, and structured evaluation can alter this relationship, but they also change the task represented by the feedback.

The reward model itself introduces ordinary generalization error. Sparse comparisons provide limited coverage of the space of possible outputs, especially when outputs are long or compositional. Since policy optimization preferentially searches for high-scoring regions, even small systematic reward errors can become behaviorally significant. This asymmetry distinguishes reward-model error from prediction error in settings where the predictor is not subsequently optimized against.

RLHF also creates institutional dependencies because evaluation criteria are translated into annotation instructions and dataset composition. The resulting behavior reflects decisions about which prompts are included, which comparisons are requested, and how conflicting judgments are aggregated. These decisions are part of the learned objective, even when they do not appear explicitly in the final reward score.

Relationship to adjacent methods

RLHF differs from ordinary supervised fine-tuning in the structure of its labels and the role of optimization. Supervised fine-tuning generally increases the likelihood of selected target outputs. RLHF instead learns a comparative evaluation function and uses that function to assess outputs generated during policy training. The distinction becomes less sharp when preference data are converted directly into a classification or likelihood objective.

Direct preference optimization and related methods estimate a policy from preferred and dispreferred responses without training a separately deployed reward model or running an explicit reinforcement-learning loop. These methods can be derived from particular regularized reward-maximization formulations. They are therefore closely related to RLHF in objective and data source, although their computational organization differs.

Reinforcement learning from AI feedback replaces some human judgments with evaluations generated by another model. Human input may remain present through principles, demonstrations, model selection, or auditing. The change concerns the immediate source of preference labels rather than the general problem of constructing a learned objective.