Neural scaling law
A neural scaling law is an empirical relationship describing how the performance of a neural network changes as the amount of training computation, model capacity, or training data varies. In contemporary machine learning, the term most commonly refers to power-law relationships between the loss of a trained model and the resources used to produce it. These relationships are statistical regularities rather than exact physical laws, and their parameters depend on the model architecture, data distribution, optimization method, and evaluation task.
Neural scaling laws became a central component of large-scale language-model research during the early 2020s. They supplied quantitative descriptions of performance trends that had previously been discussed mainly through comparisons among individual systems. Their principal use in research has been the separation of predictable resource-dependent behavior from effects caused by changes in architecture, data quality, or training procedure.
Mathematical form
For an autoregressive language model, performance is usually represented by the average cross-entropy loss on held-out text. A basic parameter-scaling relationship has the form
[ L(N)=L_{\infty}+\frac{A}{N^{\alpha}}, ]
where (L(N)) is the validation loss, (N) is the number of trainable parameters, and (L_{\infty}) represents the limiting loss under the specified data distribution. The coefficient (A) sets the scale of the reducible component, while the exponent (\alpha) determines how rapidly that component decreases as the model grows.
A corresponding relationship can be defined for the number of training tokens (D):
[ L(D)=L_{\infty}+\frac{B}{D^{\beta}}. ]
The exponents measured for neural language models are substantially smaller than one. Consequently, a fixed proportional reduction in loss requires a multiplicative increase in resources. This behavior produces smooth improvement across several orders of magnitude while also generating diminishing returns with respect to the absolute amount of computation.
A commonly used joint approximation is
[ L(N,D)=L_{\infty}+\frac{A}{N^{\alpha}}+\frac{B}{D^{\beta}}. ]
The parameter-dependent term represents limitations associated with insufficient model capacity. The data-dependent term represents limitations arising from finite exposure to the training distribution. This additive form is an approximation because model size and data volume also interact through optimization dynamics and representation learning.
Training compute for a dense Transformer is approximately proportional to the product of parameter count and token count:
[ C \approx kND, ]
where (C) denotes the number of arithmetic operations and (k) depends on conventions concerning the forward pass, backward pass, and parameter updates. Combining this constraint with the joint loss model produces compute-optimal values of (N) and (D):
[ N_{\mathrm{opt}} \propto C^{\beta/(\alpha+\beta)}, \qquad D_{\mathrm{opt}} \propto C^{\alpha/(\alpha+\beta)}. ]
These expressions describe how a fixed compute budget is divided between enlarging the network and presenting it with additional data. The resulting allocation depends directly on the empirically fitted exponents.
Early large-scale formulation
Systematic scaling studies preceded modern language models and included work on statistical learning theory, image classification, and neural machine translation. The later language-model literature differed in its use of broad parameter sweeps across model size, dataset size, and computational budget.
In 2020, Jared Kaplan and Sam McCandlish led an OpenAI study that measured scaling behavior in autoregressive Transformer models. The experiments found approximately power-law reductions in loss as parameter count, dataset size, and training compute increased. Related trends appeared across multiple textual distributions, although the fitted coefficients and limiting losses differed among evaluations.
The study distinguished raw model scaling from compute-optimal scaling. Under its experimental assumptions, the compute-efficient allocation placed a comparatively large fraction of additional computation into parameter growth. The resulting prescription implied that parameter count should increase more rapidly than the number of tokens processed during training. Several subsequent large language models followed a regime in which model capacity expanded faster than the associated training corpus.
The original analysis also established a relationship between upstream language-model loss and downstream task performance. Smooth improvement in cross-entropy did not imply that every task metric changed smoothly, because metrics based on exact answers or thresholded decisions transformed continuous probability changes into discrete outcomes. Scaling behavior at the loss level therefore remained more regular than behavior measured by individual benchmark scores.
Compute-optimal revision
A 2022 DeepMind investigation led by Jordan Hoffmann and You Watanabe re-estimated the allocation of parameters and training tokens using a larger collection of controlled training runs. The study examined model families trained under several compute budgets and fitted loss surfaces as functions of both model size and data volume. It found that many existing large language models were undertrained relative to their parameter counts.
The revised relationship assigned substantially more compute to additional training tokens than the earlier formulation had done. Within the investigated range, approximately compute-optimal scaling doubled the model size and the training-token count together when the available compute increased by a factor of four. The fitted optimum corresponded to roughly twenty training tokens per parameter for the dataset and model family used in the experiments, rather than to a universal constant applying to every training distribution.
The study’s principal demonstration was Chinchilla, a 70-billion-parameter model trained on approximately 1.4 trillion tokens. It used less inference memory than substantially larger contemporary systems while attaining lower evaluation loss under the study’s comparisons. The result separated total training compute from parameter count and showed that models with similar training costs could occupy materially different positions on the parameter–data loss surface.
This revision did not negate the existence of the earlier power laws. It altered the estimated exponents and the experimental interpretation of compute-efficient training. Differences in dataset reuse, learning-rate schedules, token budgets, and the treatment of embedding parameters contributed to the changed allocation.
Interpretation and scope
Scaling laws summarize behavior within a defined experimental regime. Extrapolation assumes that the architecture, optimizer, data-generating process, and loss definition remain sufficiently stable as scale changes. A discontinuity in any of these conditions changes either the fitted coefficients or the functional form itself.
The irreducible term (L_{\infty}) does not represent an absolute lower bound on language understanding. It is the limiting loss of the modeled setup on a particular distribution. Changes in tokenization or data curation alter the prediction problem and therefore alter the numerical value of the limit. The same distinction applies to comparisons between natural-language corpora and more structured domains such as source code.
Parameter count also provides an incomplete measure of functional capacity. In a dense network, most parameters participate in each token computation. In a mixture-of-experts model, total parameter count can greatly exceed the number activated for one token, which requires scaling analyses to distinguish stored parameters from active computation. Architectural modifications can shift a system away from a scaling curve fitted to dense Transformers without contradicting the measurements that produced the original curve.
Data volume has a similarly conditional meaning. Repeated presentation of identical tokens is not equivalent to exposure to the same number of independent observations. Duplicate documents reduce the effective diversity of a corpus, while changes in domain composition can improve one evaluation distribution and worsen another. Scaling studies consequently treat token count as an observable proxy for the quantity of training information rather than as a complete measure of it.
Emergent behavior and metric dependence
The continuous decline of predictive loss can coexist with apparently abrupt changes in benchmark performance. A task scored by exact match records no improvement until the model’s most probable answer crosses the decision boundary separating an incorrect output from the accepted output. Aggregating many such transitions can create a sharp rise in measured accuracy even when the underlying probability assignments change smoothly.
This metric effect accounts for part of the phenomenon described as emergent abilities. It does not imply that every capability reduces to a measurement artifact. Multi-step tasks can contain interacting failure probabilities, and a moderate improvement at each step can produce a larger change in the probability of completing the entire sequence. Scaling laws for aggregate loss alone do not specify the internal representations or algorithms responsible for that change.
Inverse scaling also occurs when increased scale strengthens an undesirable statistical regularity in the training distribution. Such behavior is compatible with an overall reduction in loss because a broad average can improve while performance on a narrow subset declines. The appropriate scaling relation therefore depends on the measured quantity and the distribution over which it is averaged.
Limitations of fitted laws
Power-law fits are sensitive to the range of models included in the analysis. Small networks can remain in an optimization-limited regime, while very large systems can encounter data exhaustion or infrastructure constraints. A single fitted exponent can obscure these transitions when the measurements cover only a restricted interval.
Training loss and evaluation loss also diverge under overfitting. Increasing parameter count can continue to reduce the objective on repeated training data without producing the validation improvement predicted by a data-rich scaling curve. Regularization and data augmentation alter this transition by changing the effective learning problem rather than by removing the finite-data constraint.
The relationship between pretraining loss and economically relevant inference cost introduces another distinction. Compute-optimal pretraining minimizes loss for a fixed training budget, whereas deployment-oriented optimization can favor a smaller model trained for more tokens because each later inference requires fewer operations. The preferred point on the scaling surface therefore changes when training and inference computation are combined into one resource measure.