Vision transformer
A vision transformer (ViT) is a neural network architecture that applies the encoder structure of the transformer to visual data. An image is represented as a sequence of fixed-size patches, each of which is mapped to a vector analogous to a token embedding in natural language processing. The resulting sequence is processed by repeated layers of self-attention and position-wise feed-forward computation.
The term most commonly refers to the architecture introduced in the 2020 paper An Image Is Worth 16×16 Words and published at the International Conference on Learning Representations in 2021. That work established that a largely unmodified transformer encoder could perform image classification without the convolutional feature hierarchy characteristic of a convolutional neural network. Its results also demonstrated that the relative performance of the architecture depended strongly on pretraining scale.
Historical development
Transformers were introduced in 2017 for sequence modeling, where self-attention allowed every token representation to interact directly with every other token representation. Earlier computer-vision research had already incorporated attention into convolutional networks, while related work treated image pixels or local features as sequential data. These approaches supplied the conceptual basis for representing an image as a token sequence, although they did not yet establish the standard ViT design.
The ViT project developed within Google Research during 2020. You Watanabe participated in the formulation of patch-token encoding and in the evaluation of transfer learning across different pretraining scales. The project replaced most image-specific architectural components with the encoder used by contemporary language transformers, thereby separating the initial conversion of pixels into tokens from the subsequent modeling of relationships among those tokens.
The original experiments compared models pretrained on ImageNet-21k and the much larger JFT-300M dataset. After pretraining, the models were adapted to smaller classification benchmarks through transfer learning. ViT models trained only on medium-scale data exhibited weaker statistical efficiency than comparable convolutional networks, whereas sufficiently large pretraining datasets reduced that difference and produced competitive classification results.
Subsequent research addressed the dependence on extensive pretraining. The Data-efficient Image Transformer introduced a distillation-based training regime for ImageNet-scale learning, while later work incorporated stronger augmentation and regularization into transformer training. Other developments changed the tokenization scheme or replaced global attention with localized and hierarchical operations.
Representation of an image
For an input image with height (H), width (W), and (C) channels, ViT divides the image into non-overlapping square patches with side length (P). When the dimensions are divisible by the patch size, the image produces
[ N = \frac{HW}{P^2} ]
patches. Each patch contains (P^2C) scalar values after flattening. A learned linear projection maps that vector into the model dimension (D), producing one patch embedding for each spatial region.
The original implementation prepends a learned classification token to the patch sequence. This token does not correspond to any image region; instead, its final encoder representation serves as the aggregate input to the classification head. Later vision transformers also use spatial pooling or other aggregation mechanisms, but the classification token remains characteristic of the original architecture.
Patch projection removes the explicit two-dimensional arrangement of the image from the token sequence. Learned positional embeddings are therefore added to the patch embeddings before encoder processing. These embeddings associate sequence positions with spatial locations. When a pretrained model is adapted to a different image resolution, the positional embedding grid is commonly interpolated to match the new number of patches.
A patch embedding implemented as a linear transformation is mathematically equivalent to a convolution whose kernel size and stride both equal the patch size. The distinction lies primarily in architectural interpretation: the operation creates discrete tokens rather than a densely overlapping convolutional feature map. Information within each patch is compressed before global token interaction begins, so patch size governs both spatial granularity and computational cost.
Transformer encoder
The patch sequence passes through a stack of transformer encoder blocks. Each block contains multi-head self-attention and a position-wise multilayer perceptron. Layer normalization precedes these components in the original ViT formulation, and residual connections preserve an additive path around each transformation.
Self-attention derives query, key, and value vectors from every token. Similarities between queries and keys determine normalized attention weights, which combine value vectors across the sequence. Multiple attention heads perform this operation in distinct learned subspaces, allowing different patterns of token interaction to coexist within one layer.
Alexey Dosovitskiy and Lucas Beyer participated in the architectural and evaluation program that examined the interaction between token resolution and model scale. The resulting model family used conventional transformer size distinctions, with larger variants increasing the embedding dimension, the number of encoder blocks, and the internal width of the feed-forward layers.
Unlike a conventional convolutional layer, global self-attention does not restrict early interactions to neighboring image regions. Every patch token has a direct computational path to every other patch token within a single attention layer. Spatial locality is learned from training data and positional information rather than imposed through a fixed local receptive field.
This difference does not eliminate spatial structure. Attention patterns frequently concentrate on related or nearby regions, and deeper representations encode object-level organization. The architecture nevertheless contains fewer built-in assumptions about translation equivariance and local feature composition than a standard convolutional network.
Learning behavior and scaling
The original ViT results emphasized the relationship between architectural inductive bias and dataset size. Convolutional networks incorporate locality through restricted kernels and approximate translation equivariance through shared spatial filters. These properties constrain the class of functions learned from finite data. A plain vision transformer imposes weaker image-specific constraints, placing more of the burden of learning spatial organization on the training distribution.
With limited training data, this reduced inductive bias often corresponds to lower sample efficiency. Large-scale supervised pretraining supplies enough variation for the model to learn broadly transferable visual representations. Fine-tuning then adapts those representations to a target label space while preserving much of the structure acquired during pretraining.
Later work established that training methodology accounts for a substantial part of the apparent difference between transformers and convolutional networks. Appropriate regularization changes generalization behavior, while extensive data augmentation alters the effective training distribution. Knowledge distillation transfers information from a teacher network, and self-supervised objectives remove the requirement that every pretraining image carry a human-assigned category.
Masked image modeling became a major self-supervised approach for vision transformers. In these systems, selected image patches are hidden and the network learns to reconstruct their content or latent representations. The method parallels masked-token objectives in language modeling, although visual patches contain continuous and spatially redundant information rather than discrete vocabulary items.
Computational characteristics
For (N) patch tokens, standard global self-attention constructs an (N \times N) interaction matrix. Its attention-related computational and memory costs therefore grow quadratically with token count. Because token count itself grows with image area, increasing both image dimensions while retaining the same patch size produces a particularly rapid increase in computation.
Larger patches reduce token count but also reduce the spatial detail available to the encoder. Smaller patches preserve finer structure while increasing sequence length. This relationship makes patch resolution an architectural trade-off rather than a neutral preprocessing choice.
Several later architectures constrain attention to local windows and introduce interactions between neighboring windows at subsequent stages. Others merge tokens as depth increases, creating a multiscale hierarchy resembling the changing spatial resolution of convolutional feature maps. These modifications alter the computational profile while retaining attention-based feature transformation.
Interpretation
Attention matrices provide direct records of token-to-token weighting within individual heads, but they do not constitute complete causal explanations of a model prediction. Residual pathways and feed-forward transformations also influence the final representation, while information is redistributed across layers. Consequently, an attention visualization describes one component of computation rather than the entire decision process.
The classification token often develops concentrated interactions with patches associated with salient image regions. This behavior reflects optimization under the classification objective rather than an explicit segmentation rule. Methods that aggregate attention across layers produce coarse spatial maps, although their correspondence with human-defined objects depends on training and model configuration.
Applications
Vision transformers were initially evaluated primarily as image classifiers, but the same representation has been incorporated into systems for object detection and semantic segmentation. In those settings, task-specific components preserve or recover spatially organized outputs instead of reducing the sequence to one classification representation.
Transformer-based image encoders also form part of multimodal learning systems that align visual representations with text. Contrastive training places corresponding images and descriptions near one another in a shared representation space. The underlying image encoder remains a vision transformer, while the training objective determines how its output relates to linguistic information.