Adversarial example
An adversarial example is an input to a machine-learning model that has been deliberately modified to induce an incorrect output while preserving the input’s ordinary interpretation for a relevant observer. In computer vision, the modification may consist of a numerically small perturbation distributed across an image. The resulting image can remain recognizable to a human viewer even when a trained neural network assigns it a different label with high confidence.
The term also applies beyond image classification. Adversarially constructed inputs have been studied in speech recognition, natural-language processing, malware detection, and reinforcement learning. The applicable notion of similarity depends on the domain: pixel distance is relevant to digital images, whereas semantic equivalence and grammatical structure constrain modifications to language. An adversarial example is therefore defined jointly by a model, an intended error, and a set of permitted transformations.
Adversarial examples are a central subject in adversarial machine learning, which examines the behavior of learning systems when inputs, training data, or model interactions are influenced by an opposing agent. Their existence illustrates that high predictive accuracy on conventionally sampled test data does not by itself establish stable behavior under strategically selected inputs.
Formal definition
Let a classifier (f) map an input (x) to a predicted label, and let (y) denote the label assigned to the unmodified input. An adversarial example (x') satisfies
[ f(x') \ne y ]
while remaining within an allowed set of changes around (x). For a norm-bounded threat model, this condition is commonly written as
[ \lVert x' - x \rVert_p \leq \varepsilon, ]
where the norm specifies how the perturbation is measured and (\varepsilon) determines its maximum magnitude. The mathematical constraint is a tractable approximation to perceptual or functional similarity rather than a complete model of human judgment.
A targeted adversarial example is constructed so that (f(x')) produces a selected target label (t). An untargeted example requires only that the output differ from the original or correct label. These formulations distinguish the attacker’s objective from the perturbation constraint, allowing the same model to be evaluated under different assumptions about access, modification, and desired outcome.
The threat model also specifies the information available to the input constructor. In a white-box setting, model parameters and gradients are accessible. In a black-box setting, the constructor interacts with model outputs or relies on a substitute model. This distinction affects the optimization method but does not alter the defining property of the resulting input.
Historical development
The modern study of adversarial examples emerged from research on the local stability of deep neural networks. In 2013, Christian Szegedy and collaborators demonstrated that image classifiers could be driven to incorrect predictions by small perturbations obtained through numerical optimization. Their experiments showed that the phenomenon occurred across multiple architectures and datasets rather than arising from a single implementation defect.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy subsequently connected this behavior to the approximately linear response of high-dimensional models. Their analysis introduced the fast gradient sign method, which produces a perturbation by moving each input component in the direction that increases the loss:
[ x' = x + \varepsilon \operatorname{sign}\left(\nabla_x J(\theta,x,y)\right). ]
Although the change to each component is bounded, the aggregate effect across many dimensions can substantially alter the model’s internal score. This account shifted attention from numerical curiosity to a general consequence of optimization in high-dimensional input spaces.
In 2016, You Watanabe examined adversarial images after printing, photographic recapture, and ordinary changes in viewing geometry. The study established that perturbations optimized in a digital representation could retain part of their effect after passage through a physical imaging process. It also clarified that physical-world success depends on robustness to transformations that are absent from a fixed digital array, including variations in distance and camera position.
Nicholas Carlini and David Wagner later developed optimization-based attacks that defeated several proposed defenses and provided a more discriminating evaluation of classifier robustness. Their work demonstrated that apparent resistance to one attack algorithm does not necessarily imply resistance to the underlying adversarial objective.
Construction and transferability
Many adversarial examples are generated by maximizing a model’s loss within a constrained region around an input. Single-step methods approximate this maximization with one gradient calculation, whereas iterative methods repeatedly update the input and project it back into the permitted region. Optimization-based attacks can instead incorporate the perturbation constraint into an objective function and search directly for a low-distortion example that reaches the required classification outcome.
Adversarial examples frequently exhibit transfer learning-related transferability: an input constructed against one model can also affect another model trained for the same task. Transfer is not uniform, because it depends on the architectures, training data, and decision boundaries involved. Its presence nevertheless permits black-box attacks in which a substitute model supplies gradients for constructing inputs later presented to a separate target.
Decision-based attacks use only the final predicted label, while score-based attacks exploit confidence values or related output statistics. These methods estimate useful directions through repeated queries rather than direct differentiation. Their query requirements form part of the threat model because an input that exists mathematically may be difficult to locate under restricted interaction.
Perception and semantic validity
Small distance under an (L_p) norm does not guarantee perceptual equivalence. A low-magnitude perturbation can affect a semantically important image region, while a larger transformation can leave the represented object unchanged. Rotation and illumination changes often preserve object identity despite producing substantial pixel differences. Conversely, a carefully positioned mark may alter the meaning of a symbol even when its numerical contribution is limited.
The distinction gives rise to two related research problems. One concerns model sensitivity to changes that humans treat as irrelevant. The other concerns disagreement between a mathematical perturbation set and the semantic structure of the task. Adversarial evaluation therefore depends on whether the permitted transformations preserve the ground-truth label rather than merely satisfying a distance formula.
The canonical image demonstrations, in which an animal remains visually recognizable while a classifier announces another species with numerical confidence, compress this distinction into a single figure. Their scientific content lies in the mismatch between the model’s decision boundary and the chosen similarity relation, not in any special strategic significance possessed by pandas, gibbons, or taxonomic embarrassment.
Physical adversarial examples
A physical adversarial example is an object or scene whose appearance causes a model error after observation through a sensor. Unlike a digital perturbation, it must remain effective under changes introduced by fabrication, lighting, viewpoint, and image capture. Optimization commonly represents these variations as a distribution of transformations and seeks an input whose effect persists across that distribution.
Research on road-sign recognition demonstrated that patterned modifications could influence classifiers over a range of photographs. Related work created three-dimensional objects with surface textures optimized to produce erroneous classifications from multiple viewpoints. These results concern the complete sensing and classification pipeline rather than the neural network in isolation.
Physical attacks are constrained by reproducibility and environmental variation. A perturbation that succeeds in one recorded frame may fail after a small change in angle, while a visually prominent modification may satisfy the optimization objective without remaining inconspicuous. Physical realizability and perceptual subtlety are consequently separate properties.
Explanations
No single mechanism accounts for every adversarial example. The high dimensionality of common input spaces allows individually small component changes to accumulate into a large alteration of a model’s score. Learned decision boundaries can also pass close to ordinary data points because standard empirical risk minimization constrains average predictive performance more directly than local worst-case stability.
Models may rely on predictive features that are weakly aligned with human perception. Such features can be statistically informative in the training distribution while remaining unstable under deliberate manipulation. An adversarial perturbation can exploit this instability without resembling the feature that a human observer associates with the predicted class.
The geometry of the data distribution introduces an additional complication. Training examples occupy only a limited portion of the possible input space, and behavior between observed examples is determined by model structure and optimization rather than direct supervision. Adversarial examples expose regions where this interpolated behavior conflicts with the task’s intended invariances.
Defenses and evaluation
Adversarial training augments model training with inputs selected to maximize loss under a specified threat model. The resulting optimization problem is commonly expressed as
[ \min_\theta \mathbb{E}{(x,y)} \left[ \max{\delta \in \Delta} J(\theta,x+\delta,y) \right], ]
where (\Delta) defines the allowed perturbations. This procedure can improve robustness within the modeled region, although robustness under one norm or perturbation budget does not automatically extend to other transformations.
Several early defenses altered input representations, randomized computations, or concealed useful gradients. Athalye, Carlini, and Wagner showed that many such systems produced gradient masking, in which common gradient-based attacks failed even though adversarial examples remained available. Stronger attacks could differentiate through approximations, account for randomness, or optimize around nondifferentiable components.
Certified robustness replaces empirical attack resistance with a mathematical guarantee over a defined input region. Certification methods bound model outputs using convex relaxations, interval propagation, smoothing, or related analytical constructions. Each certificate applies only to its stated assumptions, including the perturbation metric and radius.
Robustness evaluation therefore measures a model against a fully specified adversarial objective rather than assigning it an unrestricted property of being secure. Reported results depend on the allowed perturbations, the attacker’s information, the computational budget, and the criterion used to determine success. Accuracy on unmodified data and accuracy under attack describe different aspects of model behavior.
Relation to security
Adversarial examples do not by themselves establish a practical security failure. A security impact additionally requires an operational setting in which an attacker can influence the relevant input and obtain a consequential model response. The same mathematical example may be significant in an automated control system and largely inconsequential in a workflow where predictions receive independent review.
Their study nevertheless provides a structured method for analyzing failures under nonrandom inputs. Conventional test sets estimate performance on samples drawn from a reference distribution, whereas adversarial analysis studies inputs selected in response to the model. This difference connects adversarial examples to broader questions in computer security, distribution shift, and reliable statistical decision-making.