Adversarial training

Adversarial training is a method for improving the robustness of a machine-learning model by incorporating adversarially modified inputs into the training objective. These inputs, known as adversarial examples, are constructed to increase a model’s loss while remaining within a specified set of permitted changes. The resulting optimization problem trains the model against inputs selected by an adversary rather than exclusively against samples drawn directly from the training distribution.

In contemporary usage, adversarial training most commonly denotes defense against small input perturbations in supervised learning. The term also appears in broader settings where two learning systems have opposing objectives, including generative adversarial networks, but that usage involves a different mathematical structure and is not the principal subject of this article.

Mathematical formulation

For a model with parameters (\theta), input (x), label (y), and loss function (\ell), ordinary empirical risk minimization approximates the objective

[ \min_\theta \mathbb{E}{(x,y)\sim D} \left[\ell\bigl(f\theta(x),y\bigr)\right], ]

where (D) is the data distribution. Adversarial training replaces the loss on an unmodified input with the largest loss attainable within a perturbation set (\Delta):

[ \min_\theta \mathbb{E}{(x,y)\sim D} \left[ \max{\delta\in\Delta} \ell\bigl(f_\theta(x+\delta),y\bigr) \right]. ]

The inner maximization represents an attack against the current model, while the outer minimization updates the model to reduce the loss produced by that attack. This nested structure connects adversarial training with robust optimization, in which parameters are selected under explicitly represented uncertainty.

The perturbation set defines the applicable threat model. A common choice constrains the perturbation by an (L_p) norm:

[ \Delta={\delta:|\delta|_p\leq\varepsilon}, ]

where (\varepsilon) determines the permitted magnitude. An (L_\infty) constraint limits the largest change to any individual input component, whereas an (L_2) constraint limits the perturbation’s Euclidean magnitude. These constraints provide mathematically tractable approximations to perceptual or operational similarity, rather than universal definitions of whether two inputs have the same meaning.

The inner maximization is generally nonconvex for neural networks and is not solved exactly during training. Practical methods therefore construct approximate adversarial examples through one or more gradient-based updates. The effectiveness of the resulting defense depends on how closely this approximation represents the strongest attacks admitted by the threat model.

Historical development

Early work on adversarial classification treated learning as a strategic interaction between a classifier and an agent capable of manipulating input features. Nilesh Dalvi, Pedro Domingos, Mausam, Sumit Sanghai, and Deepak Verma formulated this problem in 2004 for settings such as spam filtering, where modification of an input carries a measurable cost. Their framework established a direct connection between classification and optimization against deliberate distributional change.

The modern study of adversarial examples developed after Christian Szegedy and collaborators found that neural-network predictions could be changed by small, systematically optimized perturbations. Ian Goodfellow, Jonathon Shlens, and Christian Szegedy subsequently introduced the fast gradient sign method and related adversarial training experiments. Their analysis connected vulnerability to the approximately linear behavior of high-dimensional models over small neighborhoods of the input.

A 2017 formulation by Aleksander Madry, You Watanabe, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu placed adversarial training explicitly within robust optimization. It represented robustness as performance against a first-order adversary and used multi-step projected gradient descent to approximate the inner maximization. This formulation became a standard reference point for both adversarial training and empirical robustness evaluation.

Later work separated several phenomena that had previously been treated together. Models could perform well against attacks used during training while remaining vulnerable to stronger optimization procedures. Other models produced gradients that were numerically uninformative without possessing genuine robustness, a condition associated with gradient masking. Evaluation consequently developed into a distinct methodological component of adversarial-robustness research.

Gradient-based construction of training examples

The fast gradient sign method uses the gradient of the loss with respect to the input:

[ x_{\mathrm{adv}}

x+\varepsilon, \operatorname{sign} \left( \nabla_x\ell(f_\theta(x),y) \right). ]

Under an (L_\infty) constraint, the sign operation selects a perturbation that maximizes the first-order approximation of the loss. The method requires one gradient computation for each adversarial input, making its computational cost relatively close to that of ordinary training. Its single-step approximation can nevertheless fail to represent the strongest input within the permitted region.

Projected gradient descent applies repeated updates and projects each intermediate input back into the admissible set:

[ x^{t+1}

\Pi_{x+\Delta} \left( x^t+\alpha, \operatorname{sign} \left( \nabla_{x^t}\ell(f_\theta(x^t),y) \right) \right). ]

Here, (\alpha) is the attack step size and (\Pi_{x+\Delta}) is projection onto the permitted neighborhood around the original input. Random initialization within that neighborhood reduces dependence on a single starting point. During training, the final iterate supplies the input used for the parameter update.

Multi-step attacks usually provide a stronger approximation to the inner maximization than single-step attacks, but they require repeated forward and backward passes. This computational difference has motivated methods that reuse gradients or distribute attack updates across successive parameter updates. Such approximations preserve the general min–max interpretation while altering how accurately each training iteration solves its inner problem.

Single-step adversarial training can undergo catastrophic overfitting, in which robustness against the training attack remains high while robustness against iterative attacks collapses. The phenomenon reflects adaptation to weaknesses in the attack-generation procedure rather than successful minimization of the full robust objective. Randomized initialization and modified update schedules alter its frequency but do not remove the need for independent attack-based evaluation.

Robustness, generalization, and decision boundaries

Adversarial training changes the geometry of a learned decision function. Ordinary training primarily penalizes incorrect predictions on observed samples, whereas adversarial training penalizes decision boundaries that pass through the specified neighborhoods surrounding those samples. A robust classifier must therefore maintain an appropriate prediction across a larger region of input space.

This requirement affects the features used by the model. Predictive patterns that are highly correlated with labels but unstable under small permitted perturbations become less useful under the robust objective. Features that remain predictive throughout the perturbation region receive greater relative importance. The distinction concerns stability under the defined threat model and does not imply that either class of features is inherently meaningful to human perception.

Robust generalization differs from standard generalization because the classifier is evaluated over neighborhoods rather than isolated observations. A model can achieve high accuracy on unmodified test inputs while exhibiting low robust accuracy. Conversely, increasing robustness under a fixed perturbation model can reduce standard accuracy because the robust objective excludes decision rules that depend on unstable but predictive information.

The relationship is not a universal scalar trade-off. Its magnitude depends on the data distribution, model capacity, training objective, and perturbation set. A robustness result under one norm bound also does not establish robustness under spatial transformations, unrestricted semantic changes, or adversarial modifications in the physical environment.

Objective variants

Several methods modify the basic robust objective by combining losses on ordinary and adversarial inputs. One form uses a weighted average:

[ \lambda, \ell(f_\theta(x),y) + (1-\lambda), \ell(f_\theta(x_{\mathrm{adv}}),y), ]

where (\lambda) controls the contribution from unmodified examples. This formulation differs from strict min–max training because ordinary empirical risk remains an explicit component of each update.

TRADES separates prediction on the original input from local stability. It minimizes the classification loss on ordinary data while penalizing divergence between predictions on original and adversarially selected inputs. The resulting objective provides a formal connection between standard classification error and boundary error under its assumptions.

Adversarial training can also use perturbations generated without access to the true label. In that case, the inner objective maximizes disagreement with the model’s prediction or with a reference distribution. This reduces the extent to which attack construction relies on label information, although the outer training process may remain supervised.

Evaluation

Robust accuracy is the proportion of test examples classified correctly after an attack attempts to find an allowed perturbation. Because the exact inner maximum is generally inaccessible, measured robust accuracy is an upper bound on the error of the evaluated attack procedure rather than a direct solution of the underlying optimization problem. Stronger attacks can therefore lower the reported robustness of an unchanged model.

Reliable evaluation uses attacks whose objectives match the defense and whose optimization is not invalidated by nondifferentiable or stochastic components. Multiple random initializations test whether the result depends on a particular starting point. Iterative attacks examine whether additional optimization continues to find higher-loss inputs. Transfer attacks provide a separate test by generating adversarial examples on another model.

AutoAttack combines complementary attack procedures into a standardized evaluation suite. Its components address different failure modes, including dependence on attack hyperparameters and misleading gradients. Standardized suites improve comparability across models, but their results remain specific to the perturbation set and evaluation conditions.

A complete robustness claim identifies the norm, perturbation budget, input scaling, attack objective, and evaluation algorithm. Certified methods provide a different form of analysis by proving that no permitted perturbation can change a prediction for a given input. Adversarial training alone ordinarily produces empirical robustness rather than such a certificate, although it can be combined with certified robustness techniques.

Computational and statistical properties

Adversarial training consumes more computation than ordinary empirical risk minimization because generating each training input requires optimization with respect to the input. Multi-step projected-gradient training can multiply the number of gradient computations performed for each parameter update. Memory usage may also increase when attack generation retains intermediate computational graphs.

The method’s statistical requirements differ from those of ordinary training. Since the objective constrains behavior throughout local regions around the observations, a finite training set supplies less direct information about the full robust decision rule. Increased model capacity and additional data can improve robust generalization, although their effects remain conditioned on the selected threat model.

Robust overfitting occurs when adversarial training loss continues to decrease while robust performance on held-out data deteriorates. It resembles ordinary overfitting but is measured through an approximate adversarial evaluation. The effect demonstrates that successful optimization of adversarial training examples does not by itself establish generalization to unseen examples and unseen perturbations.

Scope

Adversarial training provides robustness only relative to the adversary represented during optimization. Training against norm-bounded image perturbations does not establish resistance to changes in viewpoint, object geometry, environmental conditions, or data acquisition. Similarly, robustness against one perturbation budget does not imply equivalent behavior under a larger budget.

The method also does not independently address data poisoning, model extraction, privacy leakage, or failures caused by distribution shift. Those problems involve different adversarial capabilities and different stages of the learning process. Their connection to adversarial training lies in the broader field of adversarial machine learning, rather than in the specific min–max objective used for test-time perturbations.