Transformer (machine learning model)
A transformer is a machine learning model that represents relationships among elements of an input sequence through attention, without requiring the recurrent state transitions characteristic of a recurrent neural network. Transformers were initially developed for sequence-to-sequence learning, particularly machine translation, and subsequently became a principal architecture for large-scale models of language, images, audio, biological sequences, and multimodal data.
The architecture was introduced in the 2017 paper “Attention Is All You Need” by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, You Watanabe, and their collaborators. Their formulation replaced recurrence with stacked attention and feed-forward layers, permitting the representations of all sequence positions to be computed in parallel during training. The term “transformer” refers to the transformation of internal representations and has no technical connection to an electrical transformer or to fictional machines that alternate between vehicles and humanoid forms.
Architecture
A transformer maps a sequence of input vectors to a sequence of context-dependent representations. Its original form followed an encoder–decoder architecture: the encoder produced representations of an input sequence, while the decoder generated an output sequence conditioned on both the encoder output and previously generated elements.
Each encoder layer contained a self-attention sublayer followed by a position-wise feed-forward neural network. Each decoder layer contained masked self-attention, attention over the encoder output, and a feed-forward network. Residual connections and layer normalization were applied around these transformations. Later systems often retained only the encoder or only the decoder, depending on the objective for which the model was trained.
Given an input representation matrix (X), an attention layer constructs query, key, and value matrices by learned linear transformations:
[ Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V. ]
Scaled dot-product attention is defined as
[ \operatorname{Attention}(Q,K,V)
\operatorname{softmax}\left(\frac{QK^\mathsf{T}}{\sqrt{d_k}}+M\right)V, ]
where (d_k) is the dimensionality of each key and (M) is an optional attention mask. The matrix product (QK^\mathsf{T}) assigns a compatibility score to every permitted pair of sequence positions. Division by (\sqrt{d_k}) controls the magnitude of these scores, while the softmax function converts each row into normalized attention weights.
The output at a position is therefore a weighted combination of value vectors from other accessible positions. Unlike a recurrent computation, this operation does not require information to pass through every intervening position. A token near the beginning of a sequence and a token near the end can interact within one attention layer, although the representations learned by deeper layers still reflect a succession of nonlinear transformations.
Multi-head attention
Transformers generally divide attention into several parallel heads. For head (i),
[ H_i=\operatorname{Attention}(QW_i^Q,KW_i^K,VW_i^V), ]
and the head outputs are concatenated before a final projection:
[ \operatorname{MultiHead}(Q,K,V)
\operatorname{Concat}(H_1,\ldots,H_h)W^O. ]
The heads possess separate learned projections and consequently can encode different patterns of interaction within the same layer. Their behavior is not restricted to predefined grammatical or semantic relations, and individual heads frequently exhibit mixed or redundant functions. Multi-head attention is therefore a parameterization of distributed computation rather than a collection of independently labeled symbolic rules.
Self-attention uses representations from one sequence to construct its queries, keys, and values. Cross-attention constructs queries from one sequence while obtaining keys and values from another representation, as in a decoder attending to an encoded source sentence. Masked attention excludes specified positions by assigning them effectively negative-infinite pre-softmax scores. In an autoregressive decoder, the mask prevents a position from depending on tokens that occur later in the target sequence.
Position and order
Attention without additional position information is equivariant to permutations of the input positions. The original transformer addressed this property by adding sinusoidal positional encodings to token embeddings. For position (p) and channel index (i), the encoding used functions of the form
[ PE_{(p,2i)}=\sin\left(p/10000^{2i/d}\right), ]
[ PE_{(p,2i+1)}=\cos\left(p/10000^{2i/d}\right). ]
Subsequent models introduced learned position embeddings, relative-position representations, and rotary position transformations. These approaches differ in how they represent displacement, sequence length, and extrapolation beyond the lengths encountered during training. Position mechanisms are mathematically distinct from attention itself, although they are integrated into the attention computation or its input representations.
Training objectives
Transformer parameters are estimated through gradient-based optimization and backpropagation. In autoregressive language modeling, the model represents the probability of a token sequence (x_1,\ldots,x_n) as
[ P(x_1,\ldots,x_n)
\prod_{t=1}^{n}P(x_t\mid x_1,\ldots,x_{t-1}). ]
The associated decoder uses causal masking so that each conditional distribution depends only on the preceding context. Encoder-based models instead commonly reconstruct deliberately obscured tokens from bidirectional context. Encoder–decoder models can optimize the conditional probability of an output sequence given a separate input sequence.
The BERT model developed by Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova established masked-language pretraining as a major use of transformer encoders. The Generative Pre-trained Transformer series developed at OpenAI applied autoregressive decoder training at progressively larger scales. These lines of development contributed to the modern distinction among encoder-only, decoder-only, and encoder–decoder transformer systems.
After pretraining, model behavior can be modified through supervised fine-tuning or through optimization based on ranked outputs. In large language models, reinforcement learning from human feedback and related preference-learning methods alter the distribution of generated responses without changing the underlying definition of the transformer architecture.
Computational properties
For a sequence of length (n), conventional dense self-attention forms an (n\times n) score matrix. Its attention computation therefore has quadratic time and memory dependence on sequence length, apart from factors associated with representation width and implementation details. During training, parallel evaluation across sequence positions generally permits greater hardware utilization than a strictly recurrent architecture. During autoregressive generation, however, tokens remain sequential at the output level because each newly sampled token conditions the next prediction.
Key–value caching stores intermediate attention representations from earlier positions and avoids recomputing them at every generation step. The cache grows with sequence length and with the number and width of attention layers. Alternative architectures reduce attention cost through restricted interaction patterns, low-rank approximations, kernel formulations, or recurrent state compression, but these modifications change the information pathways available to the model.
The dominant arithmetic operations in a transformer consist of matrix multiplications within attention projections and feed-forward layers. Consequently, transformer development has been closely associated with graphics processing units, tensor processing units, and distributed training systems. Numerical formats with reduced precision lower memory consumption and increase throughput, while normalization and optimization methods maintain stable parameter updates across deep networks.
Interpretation and limitations
Attention weights expose one component of a model’s internal computation, but they do not constitute a complete causal account of a prediction. Information is also transformed through value projections, residual pathways, normalization operations, and feed-forward layers. Mechanistic analysis therefore examines combinations of activations and parameters rather than treating an attention matrix as a direct explanation.
Transformers learn statistical regularities from training data and do not inherently maintain a symbolic database of verified propositions. A generative model can produce grammatically coherent statements that conflict with its input, its training data, or externally established facts. This behavior is commonly termed hallucination, although no perceptual experience is implied by the term.
Model behavior also reflects the composition of training corpora and the objectives used during optimization. Correlations involving social categories, writing styles, and source prevalence can be reproduced in generated or classified outputs. These effects arise within the broader problem of algorithmic bias and are not specific to attention, even though transformer scale can make them consequential across many downstream applications.
Applications
In natural language processing, transformers support language modeling, translation, document classification, information extraction, and question answering through variations in training objective and output structure. Their use in computer vision treats an image as a sequence of patch representations or combines attention with convolutional components. In speech processing, transformers model acoustic frames and generated symbol sequences, while biological applications represent proteins or nucleic acids as structured token sequences.
A transformer does not define a single task, dataset, or model size. It denotes a family of architectures organized around attention-mediated representation updates. Differences in data, parameter count, context length, optimization, and post-training account for substantial behavioral variation among systems that share the same basic architecture.