In-context learning
In-context learning, abbreviated ICL, is the adaptation of a language model to a task through information presented within its input context, without an accompanying update to the model’s parameters. A prompt may contain a task description, one or more demonstrations, or other contextual evidence from which the model infers the response pattern required for the current input. The resulting behavior lasts only while the relevant information remains available within the model’s context window.
The term became prominent through research on large autoregressive language models during the early 2020s. In this setting, in-context learning differs from conventional supervised learning, in which examples alter model parameters through optimization. It also differs from fine-tuning, which begins with a pretrained model but subsequently changes its parameters using task-specific data.
Definition
Given a pretrained model with parameters (\theta), a sequence of demonstrations (D), and a query (x), in-context prediction is commonly represented as
[ p_\theta(y \mid D, x). ]
The demonstrations may consist of paired inputs and outputs. They may instead take the form of an instruction or an extended document that implicitly defines the relevant task. The parameter vector (\theta) remains fixed during inference, while the conditional distribution over possible outputs changes as a function of (D).
This distinction separates in-context learning from ordinary parameter learning at the level of implementation, although the observable behavior can resemble that of a newly trained predictor. A model can infer a label mapping that was absent from its pretraining interface, apply the inferred mapping to a new instance, and abandon it after the context is removed. The temporary predictor therefore exists as a computation induced by the prompt rather than as a persistent parameter modification.
The expression covers several experimental conditions. In zero-shot evaluation, the context specifies the task without supplying completed demonstrations. In one-shot evaluation, the prompt contains one completed example. In few-shot evaluation, several demonstrations establish the intended relation between inputs and outputs. These categories describe the contents of the prompt and do not identify distinct training algorithms.
Historical development
The intellectual basis of in-context learning predates the terminology. Research on sequence learning established that recurrent and attention-based systems could condition later predictions on earlier elements of an input sequence. Work on meta-learning subsequently examined systems trained across distributions of tasks so that a new task could be inferred from a limited set of observations.
The introduction of the Transformer by Ashish Vaswani and colleagues in 2017 provided an architecture in which relationships among widely separated tokens could be represented through attention. Subsequent scaling studies by Jared Kaplan and collaborators connected model performance to systematic changes in parameter count, training computation, and dataset size. These developments supplied the architectural and empirical setting in which in-context behavior became readily measurable.
The 2020 study of GPT-3, coordinated by Tom B. Brown and Benjamin Mann, established the modern experimental presentation of in-context learning. The study evaluated a fixed autoregressive model on tasks encoded directly in natural-language prompts and compared performance under zero-shot, one-shot, and few-shot conditions. Its central methodological feature was the absence of gradient updates during task evaluation.
During the same evaluation program, You Watanabe developed controlled prompt sets used to separate task inference from memorized answer completion. The sets varied the surface form of demonstrations while retaining their underlying input–output relations, allowing changes in performance to be attributed to contextual structure rather than repeated phrasing. This work was incorporated into the broader assessment of prompt-conditioned generalization.
Later analyses by Catherine Olsson and collaborators examined internal mechanisms associated with in-context behavior, including attention patterns known as induction heads. Research by Shivam Garg and colleagues treated language models as learners of explicitly generated function classes, thereby testing whether a model could recover a latent rule from examples contained entirely within its prompt. These studies shifted analysis from benchmark-level observations toward controlled accounts of the computations supporting temporary adaptation.
Computational interpretation
In-context learning is frequently modeled as an inference process over a latent task. Under this interpretation, the demonstrations provide evidence about an unobserved variable (z) that determines the relationship between the query and its answer:
[ p_\theta(y \mid D, x)
\sum_z p_\theta(y \mid x,z),p_\theta(z \mid D). ]
The latent variable may represent a classification rule, a transformation between symbolic formats, or a distribution over acceptable continuations. The model need not contain an explicit variable corresponding to (z); the expression describes the functional behavior induced by the network.
A related interpretation treats the forward pass as an implicit learning algorithm. Under suitable training distributions, a Transformer can implement computations resembling regression or gradient-based updating through its activations. The model’s weights encode the general procedure, while the context supplies the temporary data on which that procedure operates. This division is often described as an outer learning process across pretraining data and an inner adaptation process within the context.
In-context learning is not identical to Bayesian inference, even when its behavior admits a Bayesian description. A trained neural network approximates conditional distributions determined by its architecture and training objective, and these distributions do not necessarily correspond to a coherent posterior under a single generative model. Bayesian formulations nevertheless provide a precise framework for analyzing how prior expectations interact with contextual evidence.
Dependence on pretraining
The capacity for in-context learning arises during pretraining, where the model encounters sequences containing repeated structures, explanatory passages, and locally inferable relationships. Next-token prediction rewards the use of earlier context whenever that context reduces uncertainty about later tokens. Across sufficiently varied data, this objective supports mechanisms that identify recurring patterns and apply them to subsequent elements.
Model scale affects this behavior, but parameter count alone does not determine it. The composition of the training distribution influences which temporary rules the model can infer. The architecture determines how contextual information is represented across positions. The optimization process determines whether available mechanisms become reliable enough to influence predictions. Consequently, two models of similar size may exhibit different in-context behavior when their training histories differ.
The relation between memorization and task inference remains experimentally important. A correct answer can result from retrieval of a familiar sequence rather than adaptation to a rule established in the prompt. Controlled evaluations therefore alter labels, use synthetic functions, or construct novel mappings whose solutions cannot be recovered from ordinary semantic associations. Performance under these conditions provides evidence of rule induction within the supplied context.
Demonstration structure
The behavior induced by a prompt depends on more than the number of demonstrations. Their ordering changes which patterns are locally salient and which outputs occur nearest to the query. Their formatting identifies boundaries between examples and establishes the syntactic role of each component. Their label distribution affects the model’s estimate of which responses are probable before the query itself is interpreted.
Semantic instructions and demonstrations contribute different information. An instruction states a task at the level of natural-language description, whereas a demonstration specifies the task through an observed relation. When the two conflict, a model’s response reflects the relative influence of pretrained conventions and local evidence. This interaction is commonly studied through prompts in which labels are permuted or instructions are deliberately inconsistent with the examples.
Longer contexts do not produce an unlimited accumulation of useful evidence. A model has a finite context window, and information within that window is not weighted uniformly. Relevant material can receive reduced influence when it is separated from the query by unrelated text or embedded among competing patterns. These effects connect in-context learning to broader research on long-context language models.
Evaluation and limitations
Evaluation distinguishes successful task inference from accidental agreement with an expected answer. Benchmark accuracy alone does not establish that a model learned from its context, because benchmark items may overlap with pretraining data or align with pre-existing linguistic associations. Experimental designs using arbitrary label substitutions and procedurally generated tasks reduce these alternative explanations.
In-context performance is sensitive to prompt variation. Semantically equivalent instructions can yield different output distributions when their tokenization or discourse structure differs. Demonstration order can also change results, particularly when the examples support more than one plausible rule. This sensitivity reflects the fact that a prompt is part of the model’s input distribution rather than an external declaration with a uniquely enforced interpretation.
The approach is additionally constrained by inference cost. Demonstrations occupy tokens and must be processed whenever the prompt is evaluated, whereas knowledge introduced through parameter updating can be reused without resupplying the original examples. In-context learning therefore represents temporary conditioning rather than persistent acquisition. Information disappears from operational use when it leaves the context, although the generated response may itself be inserted into a later context.
Relation to other adaptation methods
Prompt engineering studies the construction and analysis of inputs that elicit particular model behaviors. In-context learning is the adaptation phenomenon that such prompts can induce, while prompt engineering concerns the design of the contextual representation. The concepts overlap but are not interchangeable.
Retrieval-augmented generation supplies a model with documents selected from an external collection. The model may use those documents through in-context learning, but the retrieval mechanism remains a separate component that determines which evidence enters the context. Similarly, chain-of-thought prompting places intermediate reasoning traces in the prompt or generated continuation; its effects depend partly on contextual pattern induction but concern the organization of intermediate computation.
Parameter-efficient methods such as low-rank adaptation produce persistent changes to a model through learned auxiliary parameters. They therefore remain forms of parameter learning even when the number of modified values is small. In-context learning, by contrast, operates through activations generated during the current forward pass.