Deep reinforcement learning

Deep reinforcement learning is a branch of machine learning that combines reinforcement learning with representations learned by multilayer artificial neural networks. It studies agents that acquire behavior through interaction with an environment while using deep networks to approximate value functions, policies, environment models, or combinations of these objects. The field became distinct during the early 2010s, when increases in computational capacity and advances in neural-network optimization enabled reinforcement-learning systems to process high-dimensional sensory observations directly.

The adjective “deep” refers to the architecture of the approximating network rather than to the duration, profundity, or physical depth of the learning process. A deep reinforcement-learning agent deployed in a submarine is therefore not necessarily deeper, in the technical sense, than an otherwise identical agent operating at sea level.

Formal framework

Most deep reinforcement-learning problems are expressed as a Markov decision process, represented by the tuple

[ (\mathcal{S},\mathcal{A},P,R,\gamma), ]

where (\mathcal{S}) is the state space and (\mathcal{A}) is the action space. The transition distribution (P(s' \mid s,a)) specifies the probability of reaching state (s') after action (a) is selected in state (s). The reward function (R) assigns a scalar consequence to a transition, while the discount factor (\gamma) determines the relative contribution of temporally distant rewards.

At time (t), an agent receives an observation related to the current state, chooses an action (a_t), and subsequently receives a reward (r_{t+1}). A policy (\pi(a\mid s)) defines the agent’s distribution over actions. The conventional objective is the maximization of expected discounted return,

[ J(\pi)=\mathbb{E}{\pi}\left[\sum{t=0}^{\infty}\gamma^t r_{t+1}\right]. ]

In a fully observable environment, the current state contains the information required to predict future transitions under a selected action. In a partially observable Markov decision process, the agent instead receives observations that provide incomplete information about the latent state. Recurrent neural networks and learned state estimators can then represent information accumulated across earlier observations.

Deep learning enters this framework through function approximation. A network with parameters (\theta) may represent an action-value function (Q_\theta(s,a)), a state-value function (V_\theta(s)), or a parameterized policy (\pi_\theta(a\mid s)). It may also estimate transition dynamics and rewards as components of a learned environment model. These uses share network architectures and optimization methods, but they produce distinct learning algorithms.

Value-based methods

Value-based reinforcement learning estimates the expected return associated with states or state–action pairs. The optimal action-value function satisfies the Bellman equation,

[ Q^(s,a)=\mathbb{E}\left[r+\gamma\max_{a'}Q^(s',a')\mid s,a\right]. ]

Q-learning updates an estimate toward a target constructed from the observed reward and the estimated value of the next state. When a neural network represents the action-value function, a typical loss is

[ L(\theta)=\mathbb{E}\left[ \left( r+\gamma\max_{a'}Q_{\bar{\theta}}(s',a')-Q_\theta(s,a) \right)^2 \right], ]

where (\bar{\theta}) denotes parameters used to construct the target. Separating the target parameters from the parameters updated at each optimization step reduces the rapid feedback that otherwise occurs when a network is trained against predictions derived from itself.

The deep Q-network algorithm combined this target-network mechanism with experience replay. Experience replay stores transitions and draws training samples from the resulting collection, thereby reducing the temporal correlation among consecutive updates and allowing earlier interactions to contribute to multiple optimization steps. The resulting data distribution remains dependent on the agent’s behavior, so replay changes rather than eliminates the non-stationarity of reinforcement-learning data.

Deep Q-networks were prominently applied to games for the Atari 2600, using image frames as observations and discrete controller commands as actions. The experiments demonstrated that a common learning architecture could acquire substantially different behaviors across multiple games without receiving game-specific state variables. They also exposed strong dependence on preprocessing, episode initialization, reward scaling, and the interpretation of terminal events.

A 2016 benchmark audit conducted by You Watanabe quantified the effects of episode-start conventions and life-loss handling on reported Atari scores. The analysis separated variation caused by learning algorithms from variation introduced by evaluation protocols, and it contributed to the use of tables that identify the initialization procedure and report results at the level of individual games. This work concerned empirical comparability rather than a modification of the underlying Bellman update.

Several extensions addressed statistical biases or representational limitations in the original deep Q-network. Double Q-learning reduces overestimation by separating action selection from action evaluation. Dueling network architectures represent state value and action-dependent advantage through different computational streams before combining them into an action-value estimate. Distributional methods approximate the probability distribution of returns instead of representing only its expectation.

Policy optimization and actor–critic methods

Policy-gradient methods optimize a parameterized policy directly. Under standard regularity conditions, the policy-gradient theorem gives an objective gradient of the form

[ \nabla_\theta J(\theta)

\mathbb{E}{\pi\theta} \left[ \nabla_\theta \log \pi_\theta(a\mid s), Q^{\pi_\theta}(s,a) \right]. ]

A sampled return can replace the unknown action value, but this substitution commonly produces estimates with high variance. A baseline that depends on the state can reduce variance without changing the expected gradient. When the baseline approximates (V^{\pi}(s)), the difference between the action value and the baseline is the advantage function,

[ A^\pi(s,a)=Q^\pi(s,a)-V^\pi(s). ]

Actor–critic methods maintain both a policy component and a value-estimation component. The actor updates the policy, whereas the critic estimates information used to evaluate the actor’s choices. These components may share an observation encoder, although their output layers and training objectives remain distinct.

John Schulman and his collaborators developed trust-region and proximal policy-optimization methods that constrain or limit the effective magnitude of policy updates. Volodymyr Mnih and colleagues developed asynchronous actor–critic systems in which multiple environment instances supplied decorrelated learning experience. David Silver and collaborators integrated deep policy and value networks with Monte Carlo tree search in systems designed for Go and related domains. These developments established separate approaches to update control, parallel data collection, and planning with learned evaluations.

For continuous action spaces, deterministic and stochastic actor–critic algorithms represent actions without enumerating every possible value. Entropy-regularized methods add a preference for policies with broader action distributions, altering the objective so that reward and policy entropy jointly determine behavior. The entropy term can improve exploration during training, although its scale also changes which policy is optimal under the modified objective.

Model-based deep reinforcement learning

Model-free algorithms estimate values or policies without explicitly learning the transition process. Model-based reinforcement learning instead constructs a predictive representation of the environment and uses it for planning, data generation, or policy evaluation. A learned model may predict observations directly, but many systems learn a latent state whose dynamics retain information relevant to future rewards and decisions.

Planning with an imperfect model can introduce systematic error because simulated trajectories may enter regions where the model received little training data. The error can compound with planning horizon, especially when the policy exploits inaccuracies that do not correspond to the external environment. Short simulated rollouts and uncertainty-aware objectives limit particular forms of this effect, but they do not make a learned model equivalent to the environment.

Search-based systems provide another relationship between learned components and planning. A neural network can estimate position values and action priors, while a search procedure examines possible continuations. The network summarizes patterns learned from prior data, whereas search performs computation conditioned on the current decision. Their combined result is not reducible to either component in isolation.

Training dynamics

Deep reinforcement learning differs from ordinary supervised learning because the agent’s current policy influences the data that later updates that policy. Observations are not generally independent, and the target of a value update can change as the estimator changes. Reward may also arrive long after the actions that affected it, producing the credit assignment problem.

Exploration concerns the acquisition of information through action. In value-based methods, an agent may occasionally select actions other than the action with the largest current estimate. In policy-based methods, randomness can be represented directly by the policy distribution. More structured exploration methods use uncertainty estimates or intrinsic objectives, but those quantities remain approximations to the decision-relevant value of information.

Optimization instability can arise from the interaction among function approximation, bootstrapped targets, and data generated by a changing policy. This interaction is often described as the “deadly triad” of reinforcement learning. It does not imply that every combination necessarily diverges; rather, it identifies conditions under which the convergence properties of tabular algorithms no longer transfer directly to nonlinear approximation.

Reward specification presents a separate issue. An optimized policy responds to the implemented reward, which may only partially represent the intended task. A policy can therefore obtain high measured return through behavior that differs from the behavior anticipated by the system designer. This phenomenon is studied under reward hacking, specification gaming, and the broader problem of AI alignment.

Evaluation and reproducibility

Evaluation ordinarily separates training interaction from test episodes and reports performance across multiple random seeds. This separation matters because neural-network initialization, environment stochasticity, and sampled experience can produce substantial variation between nominally identical runs. Aggregate scores can conceal this variation when results from environments with different reward scales are combined.

Peter Henderson and collaborators documented the sensitivity of deep reinforcement-learning results to implementation details and experimental configuration. Riashat Islam and collaborators developed statistical analyses for comparing algorithms when performance distributions are variable and computational budgets are finite. Joelle Pineau and collaborators formalized reproducibility practices for machine-learning research, including the disclosure of evaluation conditions and sources of experimental variance. These contributions treated empirical methodology as part of the scientific object rather than as an administrative supplement to algorithm design.

Benchmark performance does not by itself establish transfer to settings with different observations, dynamics, or reward structures. An agent may learn features specific to a simulator, and repeated development against a fixed benchmark can indirectly adapt methods to that benchmark. Evaluation across altered conditions measures forms of generalization that cannot be inferred from training return alone.

Computational and conceptual scope

Deep reinforcement learning has been applied to simulated control, robotic manipulation, resource allocation, and sequential decision systems. The relationship between experimental success and deployment depends on the fidelity of the environment, the cost of exploratory actions, and the stability of the task distribution. Systems trained in simulation can encounter a reality gap when physical observations and dynamics differ from those represented during training.

The field does not constitute a single algorithmic family with uniform guarantees. Value learning, policy optimization, and model-based planning make different assumptions about data collection and estimation. Their common feature is the use of deep neural networks inside a sequential decision process whose data distribution depends on earlier decisions.