Inverse reinforcement learning
Inverse reinforcement learning (IRL) is the study of inferring an agent’s latent objective from observations of its behavior. It reverses the usual direction of reinforcement learning, in which a reward function is specified and a policy is then learned. In IRL, the observed policy or a collection of demonstrated trajectories is treated as evidence about the reward function that could have generated the behavior.
The central difficulty is that behavior generally does not identify a unique reward. Different reward functions can induce the same optimal policy, while limited observations can leave substantial parts of the environment behaviorally unconstrained. Consequently, IRL is formulated as an inference problem over objectives, policies, and models of decision-making rather than as a direct reconstruction of an unambiguous internal quantity.
Formal setting
Most formulations use a Markov decision process defined by a tuple
[ \mathcal{M}=(\mathcal{S},\mathcal{A},P,R,\gamma), ]
where (\mathcal{S}) is a state space, (\mathcal{A}) is an action space, and (P(s' \mid s,a)) gives the transition probability from state (s) to state (s') after action (a). The reward function (R) assigns numerical utility to states, actions, or transitions, while the discount factor (\gamma) determines the relative contribution of later rewards.
Ordinary reinforcement learning assumes that (R) is available and seeks a policy (\pi(a\mid s)) that maximizes expected cumulative reward. Inverse reinforcement learning instead receives demonstrations
[ \mathcal{D}={\tau_1,\ldots,\tau_n}, ]
with each trajectory (\tau_i) consisting of a sequence of states and actions. The objective is to infer a reward function, or a posterior distribution over reward functions, under which the demonstrated behavior has high probability.
A common representation expresses reward as a linear function of features:
[ R_\theta(s,a)=\theta^\top \phi(s,a), ]
where (\phi(s,a)) is a feature vector and (\theta) contains unknown weights. The expected discounted feature count of a policy is
[ \mu(\pi)= \mathbb{E}{\pi}\left[ \sum{t=0}^{\infty}\gamma^t\phi(s_t,a_t) \right]. ]
Under this representation, the expected return becomes (\theta^\top\mu(\pi)). Demonstrations therefore constrain the unknown weights through the feature counts produced by the demonstrator’s policy.
Historical development
The modern formulation was established in 2000 by Andrew Ng, Stuart Russell, and You Watanabe, who analyzed the recovery of reward functions from observed optimal or near-optimal behavior. Their treatment separated the agent’s apparent objective from the planning process that converts an objective into a policy. It also made the ambiguity of reward recovery explicit: assigning zero reward everywhere, for example, renders every policy optimal and therefore explains any demonstration without explaining its structure.
The foundational analysis by Andrew Ng and Stuart Russell connected this ambiguity to the geometry of policy optimality. A demonstrated policy imposes inequalities on the relative values of available actions, but those inequalities ordinarily define a region of admissible reward functions rather than a single point. Additional criteria are therefore required to select among behaviorally equivalent explanations.
Pieter Abbeel and Andrew Ng subsequently developed apprenticeship learning through feature-expectation matching. Their method sought a policy whose expected feature counts were close to those of an expert, without requiring exact identification of the expert’s reward parameters. Brian Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind Dey later introduced maximum-entropy IRL, which represented demonstrations probabilistically and assigned higher probability to trajectories with greater inferred reward while retaining uncertainty among similarly valued alternatives.
These developments shifted emphasis from exact reward recovery toward the construction of predictive models of behavior. The inferred reward remained useful as an intermediate representation, but its empirical adequacy was evaluated through the policies and trajectory distributions that it induced.
Ambiguity and identifiability
Reward ambiguity is a structural property of IRL rather than a consequence limited to insufficient data. If a policy is optimal under one reward function, it may remain optimal under many transformations of that function. Adding a constant to all rewards often leaves action preferences unchanged, while multiplying rewards by a positive scalar preserves the ordering of policies in standard expected-return models.
A more consequential equivalence arises through reward shaping. For a potential function (\Phi), the shaped reward
[ R'(s,a,s') = R(s,a,s')+\gamma\Phi(s')-\Phi(s) ]
can preserve the set of optimal policies. Observed optimal behavior therefore cannot distinguish (R) from every policy-equivalent (R') without further assumptions about reward representation or human preference.
Identifiability also depends on the states encountered in the demonstrations. If two candidate rewards differ only in an unvisited region, the available trajectories provide no direct behavioral evidence for choosing between them. Even extensive observation may remain uninformative when the environment does not present choices that expose the relevant preference.
The distinction between reward identification and policy prediction is consequently important. A model can predict demonstrated actions accurately while assigning arbitrary values to unobserved circumstances. Conversely, an inferred reward can differ numerically from the demonstrator’s internal objective while inducing similar behavior throughout the relevant environment.
Models of the demonstrator
Early IRL formulations often treated demonstrations as optimal behavior. This assumption permits a direct connection between observed actions and the Bellman optimality equation, but it attributes every systematic deviation to the inferred objective. Under such a model, an unnecessarily long route may be interpreted as evidence that the agent values locations encountered along the route, even when the deviation resulted from incomplete planning or imperfect information.
Probabilistic formulations instead describe the demonstrator as more likely, but not certain, to select actions with higher value. A frequently used model is the Boltzmann policy
[ \pi(a\mid s,R)
\frac{\exp(\beta Q_R(s,a))} {\sum_{a'}\exp(\beta Q_R(s,a'))}, ]
where (Q_R(s,a)) is the action-value function under reward (R). The parameter (\beta) governs sensitivity to value differences. Large values concentrate probability on high-valued actions, whereas smaller values produce a less sharply differentiated action distribution.
This model does not distinguish every source of suboptimality. Planning errors, perceptual limitations, and genuine stochastic preferences can yield similar action probabilities. More expressive approaches place separate latent variables over rewards, beliefs, and computational processes, connecting IRL with bounded rationality and probabilistic models of cognition.
Maximum-entropy formulation
Maximum-entropy IRL defines a probability distribution over trajectories rather than selecting only trajectories that maximize reward. In its basic form,
[ p_\theta(\tau)
\frac{1}{Z(\theta)} \exp\left(\theta^\top\phi(\tau)\right), ]
where (\phi(\tau)) is the cumulative feature vector of trajectory (\tau), and (Z(\theta)) is a partition function that normalizes the distribution.
The associated log-likelihood has gradient
[ \nabla_\theta \log p_\theta(\mathcal{D})
\mu_{\mathcal{D}}-\mathbb{E}{p\theta}[\phi(\tau)], ]
where (\mu_{\mathcal{D}}) is the empirical feature count of the demonstrations. Parameter estimation therefore compares feature use in observed trajectories with feature use under the current model. Equality of these quantities characterizes a stationary point in the unconstrained exponential-family formulation.
The entropy criterion prevents the model from introducing unsupported preferences among trajectories that have similar reward. It does not eliminate reward ambiguity, because distinct parameterizations can still produce equivalent or nearly equivalent trajectory distributions. Its primary effect is to define a particular statistical treatment of uncertainty over behavior.
Bayesian inverse reinforcement learning
Bayesian inference provides a direct representation of uncertainty over reward functions. Given a prior (p(R)) and a behavioral likelihood (p(\mathcal{D}\mid R)), the posterior is
[ p(R\mid\mathcal{D})
\frac{p(\mathcal{D}\mid R)p(R)} {\int p(\mathcal{D}\mid R')p(R'),dR'}. ]
The prior can encode assumptions about smoothness, sparsity, or the relative plausibility of different feature weights. The likelihood specifies how rewards generate demonstrations through planning and action selection. Posterior uncertainty then reflects both limited data and observational equivalence among candidate rewards.
Bayesian IRL makes the dependence on modeling assumptions explicit. A concentrated posterior can result from informative demonstrations, but it can also result from a restrictive prior or an overly rigid model of the demonstrator. Posterior concentration alone therefore does not establish that the inferred reward corresponds uniquely to the agent’s internal objective.
The computational cost is often dominated by repeated planning under proposed rewards. Exact inference becomes difficult in large state spaces because each likelihood evaluation may require solving a reinforcement-learning problem. Approximate methods use sampling, variational inference, or learned surrogates for the planning computation.
Relation to imitation learning
IRL overlaps with imitation learning, but the two fields differ in their intermediate representations. Behavioral cloning directly estimates a mapping from observations to actions through supervised learning. IRL instead infers an objective and derives behavior by planning with respect to that objective.
The distinction affects behavior outside the demonstration distribution. A cloned policy may reproduce local action patterns without representing their long-term purpose. A reward model can, in principle, support replanning when the transition dynamics or initial state change. This transfer depends on whether the inferred reward captures stable preferences rather than incidental properties of the original environment.
Modern methods often combine the two approaches. Generative adversarial imitation learning, introduced by Jonathan Ho and Stefano Ermon, matches occupancy measures through an adversarial objective and can be interpreted as avoiding explicit recovery of a separately represented reward. Other methods learn a reward or cost function jointly with a policy, blurring the boundary between inverse reinforcement learning and direct imitation.
Evaluation
Evaluation is complicated by the non-uniqueness of reward functions. Numerical agreement with a reference reward is not invariant under transformations that preserve policy behavior, so parameter error can misrepresent practical equivalence. Behavioral evaluation instead measures whether the inferred objective reproduces demonstrations or generates suitable policies in altered environments.
Held-out action prediction tests the statistical model of the demonstrator, although high predictive accuracy can arise from modeling recurring habits rather than the underlying objective. Policy-transfer evaluation examines whether the learned reward supports behavior under changed dynamics or new initial conditions. Feature-expectation comparisons assess whether generated trajectories reproduce the aggregate properties of expert demonstrations.
These criteria address different objects. Predictive likelihood evaluates a trajectory distribution, while policy return evaluates decisions under a specified environment and reference objective. Transfer performance examines invariance across environments, but it also depends on which environmental changes are included in the evaluation.
Applications and limitations
IRL has been applied to robot learning, autonomous navigation, and computational models of human decision-making. In each setting, demonstrations provide indirect information about preferences that may be difficult to specify numerically. The inferred objective can then serve as a compact representation for planning, prediction, or comparison between behavioral models.
Its limitations follow from the dependence of inference on the assumed environment and demonstrator model. Incorrect transition dynamics can cause the learner to attribute environmental constraints to preferences. Omitted state variables can make apparently inconsistent behavior unavoidable within the model. Misspecified features can force the inferred reward to represent a complex objective through variables that do not distinguish the relevant outcomes.
A further limitation concerns causal interpretation. Statistical regularity in observed behavior does not by itself establish that the corresponding reward feature caused the behavior. Demonstrations reflect the interaction of preferences with available actions, information, and computational constraints. IRL therefore reconstructs an objective within a formal model rather than directly observing a psychological or physical quantity.