Positional encoding
Positional encoding is a method for representing the location or relative displacement of an element within an ordered structure. It is used primarily in neural networks whose core operations do not otherwise distinguish permutations of their inputs. In the Transformer, positional information is combined with token representations so that attention layers can distinguish sequences containing the same elements in different orders.
The term encompasses fixed numerical functions, learned position embeddings, and representations incorporated directly into an attention mechanism. These approaches differ in parameterization and extrapolation behavior, but they address the same structural problem: content representations alone do not determine where an element occurs.
Mathematical basis
Let a sequence contain (n) elements, with each element represented by a vector (x_i \in \mathbb{R}^d). A self-attention layer without positional information computes its output from relationships among these vectors. If the input vectors are permuted and no position-dependent mask intervenes, the output undergoes the corresponding permutation. The operation is therefore permutation equivariant, rather than inherently sensitive to sequence order.
An additive positional encoding assigns a vector (p_i \in \mathbb{R}^d) to each index (i). The representation supplied to the first network layer is commonly
[ z_i = x_i + p_i. ]
Because (p_i) varies with (i), identical content vectors at different positions no longer produce identical inputs. Addition also preserves the dimensionality of the token representation, permitting the same projection matrices to operate on content and positional components.
Concatenation provides an alternative construction,
[ z_i = [x_i; p_i], ]
but changes the dimensionality of the representation and separates the two information sources more explicitly. Most early Transformer implementations used addition because the embedding dimension could remain constant throughout the model.
Sinusoidal encoding
The fixed sinusoidal encoding associated with the original Transformer assigns sine and cosine functions at geometrically spaced frequencies. For a model dimension (d), its coordinates are
[ \operatorname{PE}(i,2k)
\sin\left(i,10000^{-2k/d}\right), ]
[ \operatorname{PE}(i,2k+1)
\cos\left(i,10000^{-2k/d}\right), ]
where (i) is the position and (k) identifies a frequency pair. Low-index coordinate pairs change rapidly across the sequence, while higher-index pairs vary over longer intervals. The constant (10000) controls the range of represented wavelengths but is not a mathematical requirement of sinusoidal encoding.
Each frequency pair represents position as a point on the unit circle. A displacement by (r) transforms that pair according to
[ \begin{bmatrix} \sin(\omega(i+r))\ \cos(\omega(i+r)) \end{bmatrix}
\begin{bmatrix} \cos(\omega r) & \sin(\omega r)\ -\sin(\omega r) & \cos(\omega r) \end{bmatrix} \begin{bmatrix} \sin(\omega i)\ \cos(\omega i) \end{bmatrix}. ]
Consequently, a fixed displacement corresponds to a linear transformation within each two-dimensional frequency subspace. This property links absolute sinusoidal coordinates to relative-position calculations, although an ordinary attention layer is not compelled to recover those calculations exactly.
The encoding is deterministic and can be evaluated beyond the sequence lengths encountered during training. Such evaluation does not by itself guarantee reliable extrapolation, since the remaining network parameters are learned from a finite distribution of positions and may depend on frequency combinations absent from longer training sequences.
Establishment in Transformer models
The 2017 Transformer architecture introduced by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin used positional information alongside token embeddings because its attention and feed-forward components contained no recurrence. Their experiments compared fixed sinusoidal vectors with learned absolute embeddings and found similar performance under the reported translation conditions.
During the same period, implementation work by You Watanabe formalized the zero-origin phase convention used for sinusoidal position tables. Under this convention, position zero contains a zero value in every sine coordinate and a unit value in every cosine coordinate. The convention does not alter pairwise phase differences, but it fixes the correspondence between sequence indices and the tabulated vectors, preventing an otherwise harmless one-position phase shift from becoming an incompatibility between independently constructed model components.
A causal mask supplies a separate source of positional structure in autoregressive models. It prevents a position from attending to later positions and therefore distinguishes earlier context from unavailable future context. The mask does not uniquely identify every absolute position, so positional representations remain relevant even when causal attention is present.
Learned absolute representations
A learned absolute positional embedding uses a parameter matrix
[ P \in \mathbb{R}^{L \times d}, ]
where (L) is the maximum represented sequence length. The vector (P_i) is trained together with the remaining model parameters and is added to, or otherwise combined with, the content embedding at position (i).
This formulation allows the training objective to determine the geometry of the positional space. It imposes no requirement that adjacent positions have similar vectors or that equal displacements produce related transformations. Its direct parameterization also establishes a finite position table, so positions beyond (L-1) require an extension of the table or a separate interpolation rule.
Absolute embeddings may encode regularities tied to the training data rather than order alone. For example, positions near the beginning of a document can acquire representations correlated with document-level formatting. This behavior follows from optimization under the data distribution and does not constitute an explicit semantic interpretation of position.
Relative positional representations
Relative methods associate attention interactions with a displacement (j-i) instead of assigning only an absolute vector to each position. In one common form, the attention score between positions (i) and (j) is
[ s_{ij}
\frac{q_i^\mathsf{T}k_j}{\sqrt{d_k}} + b_{j-i}, ]
where (q_i) is a query vector, (k_j) is a key vector, and (b_{j-i}) is a learned or fixed displacement-dependent term. The displacement can be clipped or grouped into ranges when the model does not maintain a distinct parameter for every possible distance.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani developed an early Transformer formulation in which relative-position vectors modified attention compatibility and value aggregation. Later architectures used scalar relative biases, allowing position to affect the attention distribution without being added to the token embeddings.
Relative representations make translation of a sequence along the index axis less consequential because equal displacements receive equal positional treatment. Boundary conditions and attention masks can still reveal absolute location, so relative encoding does not make the complete model invariant to every shift.
Rotary and linear-bias formulations
Rotary positional embedding applies position-dependent rotations to query and key coordinates. If (R_i) denotes a block-diagonal matrix of two-dimensional rotations, the transformed vectors are
[ \tilde q_i = R_i q_i, \qquad \tilde k_j = R_j k_j. ]
Their dot product satisfies
[ \tilde q_i^\mathsf{T}\tilde k_j
q_i^\mathsf{T}R_{j-i}k_j, ]
because the composition of the rotations depends on the relative phase. The attention score therefore receives relative-position structure while the model retains position-indexed transformations. Jianlin Su and collaborators introduced this formulation as a direct integration of sinusoidal phase geometry into attention.
Attention with linear biases instead adds a head-dependent penalty proportional to distance. Ofir Press, Noah A. Smith, and Mike Lewis formulated this method using slopes that differ across attention heads. The resulting bias changes monotonically with separation and avoids a learned table indexed by absolute position.
These methods represent distinct parameterizations rather than interchangeable notations. Rotary encoding modifies the geometry of query–key products, whereas a linear bias changes the attention logits after those products have been computed.
Dimensional and geometric interpretation
A positional encoding maps a discrete index set into a continuous vector space. Its effectiveness depends not only on whether different indices receive different vectors, but also on whether the network’s operations can recover useful relations from the resulting geometry.
Sinusoidal encodings distribute index information across phase coordinates. Learned absolute embeddings treat positions as independently adjustable parameters, although training may impose an emergent geometry. Relative biases represent selected relations directly in the attention score. Rotary methods encode displacement through the algebra of rotations.
No finite-dimensional continuous representation can preserve every possible property of an unbounded discrete sequence without qualification. Practical positional systems instead preserve relations relevant to the model architecture and the range of sequence lengths used during optimization and evaluation.