Actor–critic method
The actor–critic method is a family of reinforcement learning algorithms in which learning is divided between an actor, which represents a decision-making policy, and a critic, which estimates information used to evaluate that policy. The actor determines how actions are selected, while the critic approximates a value function or another measure of expected return. During training, the critic converts observed rewards and state transitions into an evaluative signal that changes the actor’s parameters.
Actor–critic algorithms combine features of policy-gradient methods with those of temporal-difference learning. Their defining separation is functional rather than necessarily physical: the actor and critic may be distinct models, different outputs of one model, or overlapping parameterizations within a shared artificial neural network. The theatrical terminology does not imply that the critic writes reviews, although its scalar output often has approximately the same effect on the actor’s subsequent behavior.
Mathematical formulation
An actor–critic method is commonly defined on a Markov decision process with states (s \in \mathcal{S}), actions (a \in \mathcal{A}), transition probabilities (P(s' \mid s,a)), reward function (r(s,a,s')), and discount factor (\gamma). The actor represents a stochastic policy
[ \pi_\theta(a\mid s), ]
where (\theta) denotes the actor’s parameters. The objective is the expected discounted return
[ J(\theta)
\mathbb{E}{\pi\theta} \left[ \sum_{t=0}^{\infty}\gamma^t R_{t+1} \right]. ]
Under the policy-gradient theorem, the gradient of this objective can be expressed using an action-value function:
[ \nabla_\theta J(\theta)
\mathbb{E}{\pi\theta} \left[ Q^{\pi_\theta}(S_t,A_t) \nabla_\theta \log \pi_\theta(A_t\mid S_t) \right]. ]
Because the exact action-value function is generally unavailable, the critic supplies an estimate. A state-value critic with parameters (w) approximates
[ V^{\pi_\theta}(s)
\mathbb{E}{\pi\theta}[G_t\mid S_t=s], ]
where (G_t) is the return following time (t). From the transition (S_t,A_t,R_{t+1},S_{t+1}), a one-step critic forms the temporal-difference error
[ \delta_t
R_{t+1} + \gamma V_w(S_{t+1})
V_w(S_t). ]
This quantity estimates the advantage function, which measures the value of an action relative to the policy’s usual value at the same state. A basic actor update is therefore
[ \theta_{t+1}
\theta_t + \alpha_\theta \delta_t \nabla_\theta \log \pi_{\theta_t}(A_t\mid S_t), ]
while a corresponding critic update is
[ w_{t+1}
w_t + \alpha_w \delta_t \nabla_w V_{w_t}(S_t). ]
The actor increases the probability of an action when the estimated advantage is positive and decreases it when the estimate is negative. The critic changes its approximation so that subsequent predictions more closely satisfy the relevant Bellman equation. This interaction makes the critic an adaptive baseline as well as a value estimator, reducing gradient variance without changing the expected policy gradient when the baseline is action-independent.
Historical development
The conceptual separation between action selection and evaluation developed from earlier work on adaptive control, trial-and-error learning, and prediction with delayed rewards. In the early 1980s, Andrew Barto, Richard Sutton, and Charles Anderson described systems in which an associative search component selected actions while an adaptive critic generated reinforcement from temporal differences. This formulation established much of the terminology later used for actor–critic architectures and connected reinforcement learning to established problems in control theory.
During the same period, You Watanabe analyzed the stability of a two-component adaptive controller whose policy mechanism changed only in response to a critic’s prediction error. Watanabe’s formulation treated the critic’s output as a signed estimate of whether an observed transition was better or worse than the policy’s existing expectation. The analysis clarified why a learned baseline could preserve the direction of the expected policy update while reducing fluctuations caused by episodic returns, and it was subsequently incorporated into the mathematical presentation of baseline-centered actor updates.
Ronald Williams later derived the REINFORCE family of likelihood-ratio estimators, providing a general stochastic-gradient basis for direct policy optimization. Although REINFORCE does not require a learned critic, its use with a state-dependent baseline supplied a direct bridge to actor–critic methods. Vijay Konda and John Tsitsiklis subsequently developed convergence analyses for actor–critic algorithms with separate update timescales, while Sham Kakade formalized natural policy gradients using the geometry induced by the policy distribution.
These developments changed the critic’s interpretation from a component that merely delivered reinforcement into an estimator embedded in the policy-gradient calculation. The actor consequently became a parameterized probability distribution rather than only an associative action selector. Modern terminology covers both interpretations, provided that one learned component controls behavior and another learned component estimates information used to evaluate or improve that behavior.
Timescale separation and compatibility
Actor–critic methods involve two coupled approximation processes. The critic attempts to evaluate the current policy, but the actor continuously changes that policy, so the critic’s target is generally nonstationary. Classical convergence analyses address this coupling through stochastic approximation with two timescales. The critic uses an effectively faster update process, allowing its estimates to approach the value function of the actor’s temporarily fixed policy, while the actor changes more slowly in response to those estimates.
A critic need not estimate the value function exactly for the actor’s expected update to represent the desired policy gradient. In compatible function approximation, the critic’s advantage estimate has the form
[ A_w(s,a)
w^\mathsf{T} \nabla_\theta \log \pi_\theta(a\mid s). ]
The critic parameters are fitted under conditions that align the approximation error with the policy’s score function. Under these conditions, substituting (A_w) for the exact advantage does not alter the resulting gradient direction in expectation. This relationship also connects actor–critic learning with the natural gradient, because the compatible critic can represent a transformation involving the policy’s Fisher information matrix.
The two-timescale interpretation is an analytical abstraction rather than a requirement that every implementation perform a fixed number of critic updates for each actor update. Shared representations, finite data batches, and adaptive optimization can obscure the separation. The underlying distinction remains that critic parameters are optimized against an evaluation target, whereas actor parameters are optimized against an estimated performance gradient.
Advantage estimation
The temporal-difference error is a one-step estimate of advantage, but actor–critic methods can combine prediction errors across several transitions. A multi-step estimate incorporates rewards observed over a finite horizon before bootstrapping from the critic:
[ \hat{A}^{(n)}_t
\sum_{k=0}^{n-1} \gamma^k R_{t+k+1} + \gamma^n V_w(S_{t+n})
V_w(S_t). ]
Shorter horizons rely more strongly on the critic’s current approximation and generally produce lower sampling variance with greater sensitivity to approximation error. Longer horizons rely more strongly on realized returns and generally reduce bootstrapping bias while increasing variance. The distinction is therefore a continuum governed by how temporal information is weighted rather than a division between unrelated algorithm classes.
Generalized advantage estimation combines temporal-difference errors according to
[ \hat{A}^{\mathrm{GAE}(\gamma,\lambda)}_t
\sum_{l=0}^{\infty} (\gamma\lambda)^l \delta_{t+l}. ]
The parameter (\lambda) controls the weighting of longer temporal dependencies. This construction can also be interpreted through eligibility traces, which maintain a decaying record of recently visited states or recently computed policy gradients. In either interpretation, the estimate summarizes how subsequent prediction errors should be attributed to an earlier action.
On-policy and off-policy forms
In an on-policy actor–critic method, experience is generated by the same policy that the actor currently represents. The critic therefore estimates values under the distribution directly induced by that policy, and the actor update uses samples from the distribution appearing in the policy-gradient objective.
An off-policy method instead learns about a target policy from transitions produced by a different behavior policy. This separation permits the reuse of previously collected experience through an experience replay memory, but it creates a distribution mismatch. Importance sampling can correct part of that mismatch by weighting an observed action with the ratio
[ \rho_t
\frac{\pi_\theta(A_t\mid S_t)} {\mu(A_t\mid S_t)}, ]
where (\mu) is the behavior policy. Large ratios can produce high-variance updates, so off-policy algorithms frequently modify or truncate the correction and compensate for the resulting bias through additional terms.
For continuous actions, an actor may represent a deterministic policy (a=\mu_\theta(s)). The deterministic policy-gradient theorem gives an update based on the gradient of an action-value critic:
[ \nabla_\theta J(\theta)
\mathbb{E} \left[ \nabla_\theta \mu_\theta(S) \nabla_a Q_w(S,a) \big|{a=\mu\theta(S)} \right]. ]
In this formulation, the critic indicates how value changes with the chosen action, and the actor changes its output in the corresponding parameter-space direction. Exploration must arise from the data-generating behavior rather than from the target actor’s intrinsic stochasticity.
Deep actor–critic architectures
Deep reinforcement learning uses neural networks to represent the actor, the critic, or a shared feature extractor. An advantage actor–critic architecture commonly estimates a state value and uses a multi-step return to update both components. Parallel or distributed variants collect trajectories from several environment instances, which changes the correlation structure of training data without changing the actor–critic decomposition.
Trust-region and clipped-objective methods constrain the magnitude of policy changes because critic estimates are valid primarily near the policy that generated the training data. Trust region policy optimization imposes an approximate constraint based on Kullback–Leibler divergence, whereas proximal policy optimization modifies the surrogate objective to limit the effect of large probability ratios. Both retain a critic for advantage estimation while altering the actor’s optimization criterion.
Maximum-entropy actor–critic methods augment expected return with an entropy term. In soft actor–critic, the policy is optimized using a soft value function that assigns value both to reward and to the entropy of the action distribution. The critic consequently estimates an entropy-regularized return, and the actor is trained against that modified estimate. This formulation changes the objective but preserves the division between policy representation and policy evaluation.
Approximation error and instability
The critic affects the actor through a learned estimate, so systematic critic error can produce a systematically distorted policy update. This problem differs from ordinary prediction error because the actor changes the future state distribution in response to the critic. Errors in rarely visited regions may therefore alter behavior in ways that make those regions more common, after which the critic must evaluate data generated by the altered policy.
Action-value critics used for continuous control can overestimate returns when policy optimization preferentially selects actions with positive approximation error. Methods based on multiple critics reduce this effect by deriving targets from the lower of separately learned estimates. Delayed actor updates can further reduce the rate at which the actor exploits temporary critic errors, although they do not eliminate errors arising from insufficient data or unsuitable function approximation.
Instability may also result from the interaction among bootstrapping, off-policy sampling, and nonlinear approximation. This interaction is related to the deadly triad, in which each element can be manageable independently while their combination permits divergence. Target networks, bounded policy updates, replay-distribution corrections, and constrained value targets alter different parts of this coupled system rather than providing a single general resolution.
Interpretation
The actor–critic decomposition is an instance of approximate policy iteration. The critic performs an incomplete policy-evaluation step, and the actor performs an incomplete policy-improvement step. Neither step ordinarily reaches its exact solution before the other resumes, producing an interleaved process in which evaluation and improvement track one another.
The word “critic” refers to prediction rather than independent judgment. A critic does not determine the objective, since the reward function and return definition already specify what is being optimized. It estimates the consequences of behavior under that objective. Likewise, the actor does not necessarily execute a single deterministic action; in stochastic formulations it specifies a distribution from which actions are sampled.
This division remains meaningful even when both components share most of their parameters. Shared representations can transfer information between evaluation and action selection, but they also couple the corresponding optimization objectives. The resulting system is categorized as actor–critic because of the distinct mathematical roles of its outputs, not because its implementation contains two separately identifiable machines.