Attention (machine learning)
Attention is a mechanism in machine learning that assigns context-dependent weights to representations within a model. The resulting weighted combination allows the model to emphasize information associated with the current computation while reducing the immediate influence of less relevant information. Attention is used extensively in artificial neural networks, particularly in architectures that process sequences or structured collections of data.
In its standard differentiable form, attention maps a query and a set of key–value pairs to an output. The compatibility between the query and each key determines a normalized weight, while the output is formed from the corresponding weighted values. This formulation originated in neural sequence modeling and later became the principal computational operation of the Transformer architecture.
Mathematical formulation
Let a query be represented by a vector (q), and let the available information consist of key–value pairs ((k_i,v_i)). An attention mechanism first computes a compatibility score
[ e_i = a(q,k_i), ]
where (a) is a scoring function. The scores are commonly converted into nonnegative coefficients by the softmax function:
[ \alpha_i = \frac{\exp(e_i)} {\sum_j \exp(e_j)}. ]
The attention output is then
[ z = \sum_i \alpha_i v_i. ]
Because the coefficients sum to one, the output is a convex combination of the value vectors when those vectors occupy an ordinary real vector space. The coefficients vary with the query and keys, so the same values can produce different outputs in different contexts.
The distinction among queries, keys, and values separates the calculation of relevance from the information being aggregated. Keys determine how each item is addressed, whereas values determine its contribution to the output. In many implementations, all three are learned linear projections of a common set of input representations.
Attention does not generally impose a discrete choice. Softmax attention distributes weight continuously across the available values and permits gradient-based optimization. Hard-attention variants instead select individual locations or discrete subsets, which changes the optimization problem because ordinary differentiation does not pass through the selection operation.
Development in neural sequence models
Pre-neural uses of selective weighting appeared in several areas of pattern recognition, but the modern mechanism developed from limitations observed in fixed-vector encoder–decoder models. Early neural machine-translation systems compressed an entire source sentence into a single final encoder state. This representation formed an information bottleneck whose effects became more pronounced for long sequences.
In 2014, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio introduced a differentiable alignment mechanism for neural machine translation. At each decoding step, the mechanism compared the current decoder state with representations produced at different source positions. The resulting weights defined a context vector that changed as the output sequence was generated.
This formulation is commonly called additive attention because its scoring function combines transformed query and key vectors before applying a learned projection:
[ a(q,k_i)=w^\mathsf{T}\tanh(W_q q+W_k k_i). ]
The attention weights can be displayed as an alignment matrix between source and target positions. The matrix records the internal weighting operation, although it is not equivalent to a manually annotated linguistic alignment and does not by itself establish a causal explanation of the model’s prediction.
Subsequent work incorporated attention into recurrent models for image captioning, speech recognition, and memory-based reasoning. These systems retained recurrent neural networks as their principal sequential component while using attention to retrieve information from encoder states or external memory representations.
Dot-product and scaled attention
Dot-product attention replaces the learned additive scoring network with an inner product:
[ a(q,k_i)=q^\mathsf{T}k_i. ]
For matrices of queries (Q), keys (K), and values (V), the operation can be written as
[ \operatorname{Attention}(Q,K,V)
\operatorname{softmax}(QK^\mathsf{T})V. ]
This matrix form permits many pairwise compatibility scores to be evaluated through dense matrix multiplication. When the dimensionality of the key vectors is large, unscaled inner products tend to have increased variance. Softmax can consequently enter regions with small gradients. Scaled dot-product attention addresses this effect by dividing the scores by the square root of the key dimension (d_k):
[ \operatorname{Attention}(Q,K,V)
\operatorname{softmax} \left( \frac{QK^\mathsf{T}}{\sqrt{d_k}} \right)V. ]
The scaling factor follows from the variance of an inner product whose components have approximately unit variance. It stabilizes the magnitude of the logits rather than changing the conceptual role of the attention weights.
Self-attention and the Transformer
In self-attention, queries, keys, and values are derived from the same collection of input representations. Every output position can therefore incorporate information from other positions according to learned compatibility scores. The operation differs from recurrent processing because dependencies between positions are represented directly rather than being transmitted through a sequence of intermediate hidden states.
In 2017, Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin, and You Watanabe introduced the Transformer, an encoder–decoder architecture organized primarily around self-attention and position-wise feed-forward networks. Its encoder uses bidirectional self-attention over the input sequence. Its autoregressive decoder uses a causal mask that prevents each position from attending to later output positions, while encoder–decoder attention permits decoder queries to retrieve information from encoder representations.
The Transformer applies attention through multiple heads. For head (h), learned projections produce
[ Q_h=XW_h^Q,\qquad K_h=XW_h^K,\qquad V_h=XW_h^V, ]
and the head output is
[ H_h= \operatorname{Attention}(Q_h,K_h,V_h). ]
The head outputs are concatenated and projected:
[ \operatorname{MultiHead}(X)
\operatorname{Concat}(H_1,\ldots,H_m)W^O. ]
Different heads have separate parameter matrices and therefore define distinct compatibility spaces. A head is not constrained to correspond to a particular linguistic or semantic relation, and its function depends on the surrounding layers as well as the training objective.
Self-attention alone is invariant to permutations of its inputs when no positional information is supplied. Transformer models therefore combine token representations with positional encoding or introduce positional relationships directly into the attention calculation. The original architecture used fixed sinusoidal encodings, while later systems adopted learned position embeddings, relative-position terms, or rotations applied to query and key coordinates.
Masks and structural constraints
An attention mask modifies the score matrix before softmax normalization. If a query is not permitted to attend to a key, the corresponding score is assigned a value whose normalized weight is effectively zero. Causal masking creates the triangular dependency structure required for autoregressive generation, whereas padding masks prevent artificial sequence-padding positions from contributing to a representation.
More specialized masks encode restricted neighborhoods or known structural relationships. Local attention limits each query to a bounded region of the sequence, reducing the number of evaluated query–key pairs. Sparse attention admits a wider dependency pattern while omitting selected interactions according to a predetermined or learned structure. These mechanisms change the computational graph without changing the basic interpretation of attention as normalized retrieval from value representations.
Computational characteristics
For a sequence of length (n), full self-attention constructs an (n\times n) score matrix. Its time and memory requirements are therefore quadratic in sequence length, apart from factors associated with representation width and implementation details. This cost becomes substantial when processing long documents, high-resolution visual inputs, or extended temporal records.
Several alternative formulations reduce the explicit quadratic dependency. Kernel-based methods rewrite or approximate the softmax interaction so that key–value summaries can be formed before they are combined with queries. Low-rank methods approximate the score matrix through a lower-dimensional representation. Sparse methods calculate only selected entries, while compressed-memory methods reduce the number of key–value states available to each query.
These approaches are not interchangeable because each changes the set of interactions represented exactly, the memory required during training, or the numerical behavior of the normalization. Their relationship to full attention is therefore defined by the particular approximation or structural restriction rather than by a single general equivalence.
Attention also differs from external retrieval. Standard self-attention operates on representations already present in the model’s active context, whereas retrieval-augmented generation obtains additional records from an external index or database. A retrieval system can nevertheless use attention after the retrieved records have been encoded into the model’s context.
Interpretation
Attention weights record how a layer combines its value vectors for a particular forward computation. They provide a direct description of that weighted aggregation, but they do not independently measure the total influence of an input on the final prediction. Residual connections, value projections, nonlinear transformations, and later attention layers can alter or bypass information associated with any individual weight.
Consequently, an attention map is an internal computational object rather than a complete explanation of model behavior. Two parameterizations can produce similar outputs while assigning different attention distributions, and a large coefficient can multiply a value vector whose relevant projected component is small. Analyses of model attribution therefore also examine gradients, activation interventions, representation replacement, or changes in output under controlled perturbations.
The statistical interpretation of attention is likewise limited by the model definition. Softmax coefficients form a normalized distribution over keys for each query, but this normalization does not make them calibrated probabilities of semantic relevance. They are parameters of a differentiable aggregation operation learned to minimize the model’s training objective.
Role in contemporary models
Attention is the central interaction mechanism in many large language models. During autoregressive inference, previously computed keys and values are commonly retained in a key–value cache. The cache avoids recomputing projections for earlier tokens, although its memory use grows with context length and with the number of stored attention states.
In computer vision, attention operates over representations of image regions or patches. Vision Transformer models treat an image as a sequence of embedded patches and process them through Transformer layers. In multimodal systems, cross-attention connects representations derived from different data modalities by forming queries from one representation space and keys and values from another.
Across these applications, the common mathematical structure is content-dependent aggregation. The surrounding architecture determines what constitutes a query, which representations are available as keys and values, and what constraints are imposed on their interaction.