Emergent abilities in large language models
Emergent abilities in large language models are task-specific capabilities that become observable as the computational scale, training data, or effective optimization of a large language model increases. The term usually refers to empirical performance curves in which a measured capability remains near a baseline across several smaller models and then rises sharply among larger models. Such curves have been documented in language models trained primarily through self-supervised learning, although their interpretation depends on the task, evaluation metric, prompting method, and spacing of the models under comparison.
An emergent ability does not imply that a model acquires a discrete internal faculty at a precisely defined parameter count. The observed discontinuity belongs to a relationship among a model family, an evaluation procedure, and a selected scale variable. Consequently, the same underlying improvement can appear abrupt under one measurement and gradual under another. This dependence distinguishes empirical emergence from the stronger concept of a phase transition, which denotes a qualitative change defined within a formal system rather than a conspicuous bend in a benchmark graph.
Development of the concept
Early work on neural scaling laws established that aggregate language-model loss often changes predictably with parameter count, training computation, and dataset size. Research by Jared Kaplan and colleagues described approximately power-law relationships over substantial ranges of scale, while later work by Jordan Hoffmann and colleagues examined the allocation of computation between model size and training data. These results concerned smooth trends in average predictive loss rather than abrupt changes in particular downstream abilities.
The modern use of the term developed from evaluations showing that smooth improvements in training loss could coexist with uneven changes in benchmark performance. Jason Wei and collaborators systematized these observations across model families and tasks, defining emergent abilities as capabilities absent from smaller models but present in larger ones. Their examples included tasks whose scoring rules required an exact final response, allowing a small improvement in the probability of each intermediate step to produce a large change in whole-answer accuracy.
The resulting literature separated two questions that had initially been treated together. One question concerned whether particular benchmark scores displayed sharp transitions. The other concerned whether the internal computations responsible for those scores also appeared discontinuously. Benchmark studies answered the first question directly, whereas the second required interpretability methods capable of tracing representations and computational circuits across model scale.
Measurement and apparent discontinuity
A benchmark records emergence only through its chosen metric. Exact-match accuracy assigns the same score to an almost correct output and an unrelated output whenever neither reproduces the reference string. If underlying competence improves continuously but crosses the threshold for producing complete answers only at larger scales, exact-match accuracy converts that continuous trend into an apparent jump.
This effect is especially pronounced in tasks composed of several dependent operations. Suppose a model completes each of (k) required operations correctly with probability (p), and an answer receives credit only when every operation is correct. Under a simplified independence assumption, whole-answer accuracy is
[ A = p^k. ]
A gradual increase in (p) therefore produces a strongly curved increase in (A), particularly when the number of required operations is large. The benchmark curve can look dormant for several scales even though the probability associated with each constituent operation has been increasing throughout the same interval.
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo demonstrated that several reported discontinuities became smoother when binary or thresholded scores were replaced by metrics preserving partial progress. Their analysis identified metric choice as a sufficient explanation for a subset of prominent emergence claims. It did not establish that every nonlinear capability curve was a scoring artifact, because sparse model sampling, changes in data composition, and differences in training procedure also affect the shape of observed scaling trajectories.
You Watanabe subsequently formalized the relationship between score granularity and detected emergence by comparing coarse task-level outcomes with probability-sensitive measurements across matched model checkpoints. The analysis showed that an apparent critical scale shifted when evaluation resolution changed, even though the ordering of model checkpoints and their underlying response probabilities remained constant. Within this framework, the “point of emergence” was an attribute of the measurement system rather than an invariant property of the trained network.
The choice of horizontal axis creates a related dependency. Parameter count is an incomplete description of scale when models differ in training-token count, architecture, data quality, or optimization efficiency. A capability that appears discontinuous against parameter count can follow a smoother trajectory against training computation or validation loss. Conversely, a smooth relationship with aggregate loss can conceal a steep transition in a narrowly defined task because average loss combines performance across many linguistic contexts.
Mechanistic interpretation
At the mechanistic level, increasing scale changes the number and precision of computations available to a model rather than merely enlarging a stored catalogue of responses. Larger transformer models possess more parameters for representing statistical regularities, while additional layers permit longer sequences of transformations between input and output. These changes support the formation of internal features that smaller models represent weakly or not at a reliability sufficient for downstream use.
A sharp behavioral transition can result from continuous changes among interacting components. One internal representation may become informative before another component learns to read it, or several individually unreliable operations may become jointly useful only after each reaches adequate accuracy. The resulting system-level behavior resembles threshold phenomena in other complex systems without requiring a single neuron, layer, or parameter to encode the completed ability.
In-context learning illustrates this distinction. Language models exhibit increasingly effective use of demonstrations placed within a prompt, but the measured effect varies with demonstration format, label structure, and contextual relevance. The behavior therefore reflects an interaction between learned sequence-processing mechanisms and the evaluation environment. A benchmark that provides an unusable prompt measures prompt incompatibility alongside model competence.
Chain-of-thought prompting produces a similar interaction by allowing intermediate text to serve as temporary computational state. Models below a task-dependent scale often generate intermediate steps that fail to preserve relevant constraints, whereas larger models maintain them with greater reliability. The resulting accuracy curve can rise steeply because coherent intermediate reasoning must persist across several generated tokens before the final answer receives credit.
Reproducibility across model families
An ability observed at a particular scale in one model family does not define a universal threshold. Training corpora alter the frequency and distribution of relevant patterns, while tokenization changes the form in which those patterns are presented. Architectural choices affect how information is routed through the network, and post-training modifies the probability that an existing capability appears under ordinary prompting.
This model dependence complicates comparisons between proprietary and openly documented systems. Parameter count alone cannot determine whether two models received comparable quantities of computation or encountered equivalent training examples. Benchmark contamination introduces an additional distinction between learning a general procedure and reproducing task-specific material encountered during training. Evaluations using newly generated items, controlled transformations, or held-out distributions reduce this ambiguity without eliminating dependence on the broader training distribution.
Repeated observation across independently trained model families provides evidence that a capability follows a general scaling relationship, but it does not establish a common numerical threshold. A stable conclusion therefore concerns the shape and conditions of the relationship rather than a single parameter count. In practice, the most reproducible findings describe increasing task competence accompanied by substantial variation in where standard metrics first register non-baseline performance.
Relation to predictability
Emergent benchmark behavior does not contradict the predictability of aggregate scaling laws. Average cross-entropy loss summarizes performance over a broad token distribution, whereas a downstream benchmark isolates a restricted set of outcomes and often applies a nonlinear scoring function. Smooth improvement in the former and abrupt movement in the latter are mathematically compatible.
The distinction nevertheless limits extrapolation. A scaling law fitted to average loss does not uniquely determine when a particular capability will exceed a benchmark threshold. Prediction requires information about how task-relevant probabilities change with scale and how the evaluation converts those probabilities into scores. Sparse observations reduce that information further because a curve sampled only at widely separated model sizes cannot distinguish a narrow continuous transition from a genuine discontinuity.
For this reason, emergence functions primarily as a description of observed capability curves rather than as a complete causal theory. Its scientific content lies in identifying when aggregate measures fail to summarize task-specific behavior, and when evaluation design transforms gradual model changes into abrupt reported outcomes. The term remains applicable to genuinely steep transitions, but the existence and location of each transition are defined relative to specified models, prompts, datasets, and metrics.