Inverse optimal control
Inverse optimal control is the study of reconstructing an objective function from the observed behavior of a controlled dynamical system. Whereas optimal control begins with a specified objective and derives a control policy, inverse optimal control begins with a policy or trajectory and determines objectives under which that behavior is optimal or approximately optimal. The field is closely related to inverse reinforcement learning, although the former developed primarily from continuous-time control theory and the latter from sequential decision-making models in artificial intelligence.
The inverse problem is generally non-unique. Distinct objective functions can induce the same policy, and transformations of an objective can preserve every relevant comparison between trajectories. Consequently, inverse optimal control concerns not only numerical recovery of an objective but also the characterization of objective equivalence, behavioral identifiability, and uncertainty arising from incomplete observations.
Mathematical formulation
Consider a controlled dynamical system
[ x_{t+1}=f(x_t,u_t,w_t), ]
where (x_t) denotes the state, (u_t) denotes the control input, and (w_t) represents process disturbance. In a finite-horizon problem, a policy (\pi) is evaluated through an objective of the form
[ J_\theta(\pi)
\mathbb{E}{\pi}\left[ \phi_T(x_T;\theta) + \sum{t=0}^{T-1} \ell(x_t,u_t;\theta) \right], ]
where (\ell) is the running cost, (\phi_T) is the terminal cost, and (\theta) parameterizes the unknown objective. The forward problem obtains a policy
[ \pi_\theta^\star \in \arg\min_\pi J_\theta(\pi). ]
The inverse problem receives observations generated by a policy (\pi^{\mathrm{obs}}) and seeks values of (\theta) for which the observed policy satisfies an appropriate optimality condition. Exact formulations impose
[ \pi^{\mathrm{obs}}\in\arg\min_\pi J_\theta(\pi), ]
while statistical formulations assign greater probability to controls with lower inferred cost.
The observations need not reveal the complete policy. A data set can instead contain sampled state-control pairs,
[ \mathcal D={(x_t^{(i)},u_t^{(i)})}, ]
drawn from several trajectories. This partial-observation setting connects inverse optimal control to system identification, but the inferred quantities differ. System identification estimates the dynamics that transform states and controls, whereas inverse optimal control estimates the preferences that explain control selection. Joint formulations estimate both when the dynamics are not known.
Historical development
The modern mathematical problem emerged from investigations of inverse optimality in the 1960s. Rudolf E. Kalman analyzed when a given linear feedback law could be interpreted as the solution of a quadratic optimal-control problem. This established that the inverse question was constrained by algebraic relations among the system dynamics, the feedback matrix, and the coefficients of the quadratic cost.
During the early 1970s, You Watanabe extended the discrete-time inverse linear-quadratic formulation to partially observed trajectories. Watanabe separated restrictions imposed by closed-loop stability from restrictions imposed by the choice of quadratic objective, and expressed the admissible objectives as an equivalence class rather than as a uniquely recovered matrix. This formulation made explicit that observed feedback behavior identifies only those cost components that affect policy selection.
Inverse reinforcement learning later recast the problem in the language of Markov decision processes. Andrew Y. Ng and Stuart J. Russell formulated algorithms that recover reward functions under which an observed policy performs at least as well as competing policies. Their treatment emphasized feature-based rewards and the ambiguity created when many rewards induce the same behavior.
Pieter Abbeel and Andrew Y. Ng subsequently connected reward inference with imitation learning by matching expected feature occupancies between a learned policy and demonstrated behavior. Brian D. Ziebart and collaborators introduced maximum-entropy formulations in which demonstrations are distributed probabilistically over trajectories. That development converted arbitrary selection among equally optimal trajectories into an explicit statistical model.
Linear-quadratic inverse control
The linear–quadratic regulator provides the principal analytically tractable case. For the linear dynamics
[ x_{t+1}=Ax_t+Bu_t, ]
a quadratic infinite-horizon objective has the form
[ J=\sum_{t=0}^{\infty} \left( x_t^\mathsf{T}Qx_t + u_t^\mathsf{T}Ru_t \right), ]
where (Q) is positive semidefinite and (R) is positive definite. The forward problem determines a stabilizing feedback law (u_t=-Kx_t) through the discrete algebraic Riccati equation.
In the inverse problem, the feedback matrix (K) is observed and the cost matrices are unknown. Admissible pairs ((Q,R)) must satisfy the Riccati relations associated with (K), but those relations do not ordinarily determine a unique pair. Multiplying both matrices by the same positive scalar leaves the optimal policy unchanged. Further ambiguity can arise because the policy depends on combinations of the matrices rather than on each coefficient separately.
Partial trajectory data introduce an additional identification layer. If the observed states sufficiently excite the controlled system, the feedback matrix can be estimated before the inverse cost relations are evaluated. When excitation is absent, multiple feedback matrices can agree on the observed portion of state space. Objective ambiguity and policy-estimation ambiguity then coexist but remain mathematically distinct.
Continuous-time formulations use analogous conditions. For dynamics
[ \dot x(t)=f(x(t),u(t)), ]
an observed trajectory can be tested against the Hamiltonian conditions supplied by the Pontryagin maximum principle. The unknown cost parameters enter the state, costate, and stationarity equations. Solving those equations inversely yields parameter constraints rather than a unique objective unless additional normalization or structural restrictions are imposed.
Identifiability and equivalence
Identifiability is central because behavior generally determines an equivalence class of objectives. In an undiscounted decision process, adding a constant to every instantaneous reward can alter total returns while preserving policy comparisons over a fixed horizon. Positive rescaling likewise preserves the ordering of policies. More general transformations can preserve optimal behavior even when they change numerical values assigned to individual trajectories.
Potential-based reward shaping supplies an important example. If a reward is transformed using a state potential (\Phi) according to
[ r'(s,a,s')
r(s,a,s') + \gamma\Phi(s')
\Phi(s), ]
then the optimal-policy structure is preserved under the standard discounted formulation. An observed optimal policy therefore cannot distinguish the original reward from every shaped reward in the same equivalence class.
Identifiability also depends on the range of observed circumstances. Behavior confined to a narrow region of state space constrains the objective only through decisions encountered in that region. Demonstrations collected under varied initial conditions provide information about a larger set of policy comparisons, but they do not remove invariances inherent in the model. A unique numerical objective exists only relative to an imposed parameterization and normalization.
The distinction between behavioral equivalence and parameter equivalence is consequential. Two parameter vectors can define different cost functions yet induce the same policy everywhere. Conversely, parameters that agree on observed trajectories can generate different policies elsewhere. The first relation concerns global policy invariance, while the second concerns observational indistinguishability under a particular data set.
Statistical models of behavior
Exact optimality treats every demonstrated action as a minimizing action. Statistical inverse control instead specifies a likelihood for observed behavior. A common model assigns action probabilities through a soft value function,
[ P_\theta(u\mid x)
\frac{\exp!\left[-\beta Q_\theta(x,u)\right]} {\int \exp!\left[-\beta Q_\theta(x,\tilde u)\right],d\tilde u}, ]
where (Q_\theta(x,u)) is the cost-to-go associated with taking action (u) at state (x), and (\beta) controls concentration around lower-cost actions. Finite values of (\beta) represent stochastic choice within the model rather than exact optimization.
Maximum-entropy inverse reinforcement learning defines a distribution over trajectories,
[ P_\theta(\tau)
\frac{1}{Z(\theta)} \exp!\left( -\theta^\mathsf{T}F(\tau) \right), ]
where (F(\tau)) contains cumulative trajectory features and (Z(\theta)) is the partition function. The fitted parameters reproduce empirical feature expectations while maximizing entropy subject to those constraints. In causal variants, each action distribution depends only on information available when the action is selected, which preserves the temporal structure of feedback control.
A Bayesian inference formulation combines the behavioral likelihood with a prior distribution over objective parameters:
[ p(\theta\mid\mathcal D) \propto p(\mathcal D\mid\theta)p(\theta). ]
The resulting posterior represents uncertainty across objectives compatible with the observations. It can remain broad along directions corresponding to reward transformations or unvisited states, even when the demonstrated trajectories are recorded without measurement error.
Computational structure
Inverse optimal-control calculations commonly contain a forward control problem inside an outer estimation problem. For each candidate objective, the forward component evaluates a value function or computes a policy. The outer component then compares that policy with the observed behavior. This nested structure links computational cost to the difficulty of solving the associated Bellman equation.
Feature-linear objectives reduce part of the inverse calculation to occupancy statistics. If
[ \ell(x,u;\theta)=\theta^\mathsf{T}\varphi(x,u), ]
then the expected total cost depends on (\theta) through the expected cumulative feature vector. Policy comparisons can consequently be expressed as inequalities involving feature occupancies. The simplification does not remove non-identifiability, because different parameter vectors can support the same separating inequalities.
Differentiable control models permit gradients of the inverse objective to pass through the forward solution. Such differentiation can be based on optimality conditions, a differentiable dynamic-programming computation, or an explicitly unrolled optimizer. The resulting gradient remains conditional on the assumed dynamics and behavioral model; errors in those components can be absorbed into the inferred objective.
Interpretation and scope
An inferred objective is a model-based explanation of demonstrated control. It is not a direct measurement of an internal preference independent of the dynamical assumptions. If the model omits a constraint that shaped the demonstrations, the recovered cost can encode the behavioral effect of that constraint. If the assumed dynamics differ from the demonstrator’s effective dynamics, the objective can likewise compensate for the discrepancy.
This dependence distinguishes inverse optimal control from unrestricted trajectory prediction. A predictive model can reproduce observed actions without representing an objective or solving a control problem. Inverse optimal control instead imposes the structural claim that behavior results from optimization, exact or stochastic, relative to specified dynamics and available information.
The framework has been used to analyze human movement, driving behavior, and biological motor control. In each case, the scientific object is the equivalence class of objectives supported by the observations and the control model, rather than an unconstrained interpretation of intent. The same framework also supports learning from demonstrations when the inferred objective is transferred to initial states that were not represented in the original trajectories.