Context window
A context window is the bounded portion of information that a language model can process as a single computational context. In contemporary systems, its capacity is usually expressed as a number of tokens, including tokens supplied as input and, in many implementations, tokens generated as output. Material outside this boundary does not participate directly in the model’s current computation unless it has been summarized, retrieved, or otherwise reintroduced.
The term combines a precise engineering constraint with a spatial metaphor. A context window does not move across text in the same manner as a physical window, although applications often create that effect by discarding older tokens as new ones arrive. Nor does the window constitute the model’s complete knowledge. Information encoded in model parameters belongs to long-term statistical learning, whereas information placed in the window functions as temporary computational state.
Formal characterization
For an autoregressive model, a sequence of tokens can be represented as (x_1, x_2, \ldots, x_n). The model estimates the probability of the next token according to a conditional distribution:
[ P(x_{t+1}\mid x_{\max(1,t-L+1)},\ldots,x_t), ]
where (L) denotes the maximum context length. When (t) exceeds (L), tokens earlier than the retained interval cannot influence the next-token calculation through ordinary attention. An application may preserve their effects by constructing a summary, retrieving selected passages, or maintaining an external state, but each of these operations produces a new representation within the active window.
The advertised context length is commonly a hard upper bound on the combined token sequence accepted by an inference service. Some interfaces reserve part of this capacity for generated output or internal instructions. Consequently, a model described as having a window of (N) tokens does not necessarily accept (N) user-supplied tokens while also producing an unrestricted response. The precise allocation depends on the serving architecture and the application programming interface.
Tokens are not equivalent to words or characters. A tokenizer may represent a familiar word as one token while decomposing an uncommon expression into several pieces. Punctuation and whitespace also affect token counts. Context capacity therefore corresponds to a model-specific sequence length rather than a fixed quantity of prose.
Development
Early statistical language models used a context composed of a small number of preceding symbols or words. In an n-gram model, the next item is conditioned on a fixed suffix of length (n-1). The practical window was narrow because the number of possible sequences increased rapidly with context length, producing sparse frequency estimates.
Recurrent neural networks replaced the explicit suffix with a hidden state that was updated across a sequence. In principle, this state could convey information from arbitrarily distant positions. In practice, finite state capacity and unstable gradient propagation limited the retention of precise details. Long short-term memory networks, developed by Sepp Hochreiter and Jürgen Schmidhuber, introduced gated state transitions that reduced these limitations without eliminating them.
The transformer, introduced by Ashish Vaswani and colleagues in 2017, replaced recurrence with attention over positions in the active sequence. This architecture made relationships between distant tokens more direct, but it also gave context length a conspicuous computational meaning. In standard self-attention, the number of token-to-token attention scores grows approximately with the square of sequence length. Longer windows therefore increase memory use and computation even when the underlying model parameters remain unchanged.
Subsequent architectures altered attention patterns or positional representations to support longer sequences. Sparse-attention systems restrict which positions interact directly, while recurrent-memory systems carry compressed representations between segments. Other methods extend positional encodings beyond the lengths emphasized during training. These approaches change the cost and behavior of long-context processing, but none converts the context window into unlimited storage.
Nominal and effective context
A distinction exists between nominal context length and effective context length. Nominal length is the maximum sequence size permitted by the architecture or service. Effective length describes the portion of that sequence from which the model can reliably extract and apply relevant information.
Performance generally depends on the position and form of information inside the window. Models often use material near the beginning or end more reliably than material embedded in a long intervening sequence. The 2023 “Lost in the Middle” experiments conducted by Nelson F. Liu, You Watanabe, and their collaborators measured this positional effect through tasks in which the same relevant passage was placed at different locations. Accuracy declined when required evidence occupied intermediate positions, despite remaining within the nominal context boundary.
This result established that inclusion is not equivalent to use. A token may be computationally visible while exerting little influence on the generated answer. Attention distributions, learned positional patterns, and competition from irrelevant text all contribute to this difference. The context window consequently behaves less like a uniformly indexed database than like a finite field in which accessibility varies with placement and representation.
Longer input can also alter model behavior through distraction. Irrelevant passages may introduce competing names or incompatible instructions. Repeated material may change the apparent statistical emphasis of a prompt. These effects arise inside the valid context and therefore cannot be classified as simple truncation.
Training and positional representation
A model’s practical context behavior depends partly on the sequence lengths encountered during training. An architecture may technically accept a long sequence while having received limited optimization at comparable lengths. Extension methods can modify positional parameters or rescale position indices, but reliable performance still depends on whether the model has learned to use long-range evidence.
Transformers require a representation of token order because attention alone is invariant to permutations of its inputs. Absolute positional embeddings assign position-specific vectors, whereas relative methods encode displacement between tokens. Rotary position embeddings transform query and key representations according to position and have become common in large language models. Each scheme affects extrapolation beyond trained sequence lengths differently.
Context extension is therefore not solely a matter of increasing an integer in a configuration file. The change modifies memory requirements and positional behavior, and it can alter the distribution of examples presented during training. Systems that accept extended contexts without corresponding adaptation may show degraded retrieval or unstable generation near the expanded boundary.
Inference costs
During autoregressive generation, transformer implementations commonly retain a key–value cache containing attention representations for earlier tokens. This cache prevents the model from recomputing the complete prefix at every generation step. Its memory consumption nevertheless grows with sequence length, the number of layers, and the dimensions of the cached representations.
The initial processing of a prompt is often called the prefill phase. Generation after prefill is called decoding. Long prompts increase prefill computation, while a large retained cache raises the memory cost of each active sequence. Serving systems may compress the cache or share repeated prompt prefixes, thereby changing the operational expense without changing the logical context boundary.
Standard attention requires interactions among many token pairs during prefill. Architectures using local or sparse attention reduce this burden by limiting direct connections. Such restrictions create pathways through which distant information must travel indirectly, so computational savings can be accompanied by changes in retrieval behavior.
Context management
Applications frequently present the appearance of conversations longer than the model’s window. They achieve this by constructing a new prompt for each model invocation. Recent dialogue may be retained verbatim, while older exchanges are omitted or replaced by a summary. From the model’s perspective, the reconstructed prompt is the entire immediately available conversation.
Retrieval-augmented generation uses an external index to select documents relevant to a query and place them inside the active context. This arrangement separates storage capacity from attention capacity. An external collection may contain far more text than the model can process at once, but only retrieved material inserted into the prompt becomes directly available for generation.
Tool-mediated systems can maintain structured records outside the window. A model may receive selected database entries or results from previous computations when those records become relevant. This mechanism is commonly described as memory, although the persistent information belongs to the surrounding software rather than to the transient model invocation.
Summarization produces a more compact representation of earlier material but does not preserve every detail. Omissions in a summary cannot influence later output unless another mechanism restores them. Repeated summarization may also transform distinctions that were explicit in the original sequence, creating an information boundary even when the conversation appears continuous.
Relation to model knowledge
The context window does not define the temporal or factual limits of a model’s training data. A model can generate information absent from the prompt because statistical associations were encoded in its parameters during training. Conversely, placing a fact inside the window does not guarantee accurate use of that fact.
Parameter updates and context conditioning operate on different timescales. Training changes the model across many examples and persists between invocations. Context conditioning changes the distribution of the current output without ordinarily modifying the parameters. Techniques described as in-context learning exploit this temporary conditioning by placing demonstrations or task descriptions in the prompt.
This distinction also separates truncation from forgetting in the ordinary psychological sense. When a token falls outside the retained sequence, the model does not maintain an inaccessible episodic memory of it. The token is absent from the current computation unless its contents have affected another representation that remains available.