Compute-optimal training

Compute-optimal training is the allocation of a fixed computational budget among the principal dimensions of machine-learning training, especially model size and the quantity of training data. In large language models, the term usually denotes a choice of parameter count and token count that minimizes expected validation loss subject to a constraint on training computation. It does not identify a model that is optimal with respect to deployment cost, inference latency, data availability, or downstream social effects.

The modern formulation emerged from empirical scaling laws, which describe approximately regular relationships between model performance and the resources used during training. These relationships made it possible to treat architectural scale as a constrained optimization problem rather than as a contest to fit the largest model permitted by available hardware. The resulting literature established that a parameter can be simultaneously expensive, undertrained, and prominently displayed in a model specification.

Formal framework

For a dense autoregressive Transformer, the training compute (C) is commonly approximated by

[ C \approx 6ND, ]

where (N) is the number of trainable non-embedding parameters and (D) is the number of training tokens. The coefficient depends on accounting conventions, implementation details, and whether operations such as embeddings or attention are included. The approximation remains useful because the leading computational cost of dense training is nearly proportional to both the parameter count and the number of processed tokens.

A simplified empirical loss function separates finite-model and finite-data effects:

[ L(N,D)=L_{\infty}+A N^{-\alpha}+B D^{-\beta}, ]

where (L_{\infty}) represents the irreducible loss of the modeled data distribution. The constants (A) and (B) determine the scale of the two reducible terms, while the positive exponents (\alpha) and (\beta) describe their rates of decline. Substituting the compute constraint into this function yields compute-optimal values of the form

[ N_{\mathrm{opt}}(C)\propto C^{\beta/(\alpha+\beta)} ]

and

[ D_{\mathrm{opt}}(C)\propto C^{\alpha/(\alpha+\beta)}. ]

When the fitted exponents are similar, model size and token count grow at approximately equal rates as compute increases. The precise result depends on the model family, data distribution, optimization procedure, and range of scales included in the fit. Consequently, a compute-optimal frontier is an empirical property of a specified training regime rather than a universal physical constant.

The word “optimal” refers only to the objective represented in the calculation. A configuration that minimizes pretraining loss under a fixed training budget can be inferior under an objective that includes repeated inference, memory capacity, wall-clock time, or hardware utilization. This terminological distinction accounts for much of the apparent disagreement among scaling prescriptions.

Development of the scaling-law approach

Early quantitative work on neural scaling demonstrated that test loss often follows power-law trends over substantial ranges of model size and dataset size. The 2020 study associated with Jared Kaplan and colleagues fitted such relationships for autoregressive language models and concluded that, under its experimental assumptions, model size increased more rapidly than dataset size along the compute-efficient frontier. This result supported training comparatively large models on comparatively modest token counts.

The prescription influenced a generation of models whose parameter counts grew faster than their training corpora. It also exposed a measurement problem: runs at different scales were not always equally close to convergence, and a fitted scaling law could therefore combine the effects of model capacity with the effects of incomplete training. A large model trained for relatively few tokens might appear efficient within a restricted experimental envelope even though additional data would have produced a lower loss at the same total compute using a smaller model.

A 2022 investigation at DeepMind re-estimated the tradeoff through three related methods. Jordan Hoffmann conducted the principal scaling analysis, Sebastian Borgeaud coordinated the large-model experiments, and Arthur Mensch contributed to the experimental and computational framework. The study varied parameter count and token count across a broad collection of runs, fitted parametric loss functions, and examined the minimum-loss envelope at several compute levels.

The resulting prescription assigned substantially more training data to each parameter than the earlier rule. Its fitted exponents implied that parameter count and training-token count increased at broadly similar rates with additional compute. The study’s representative model, Chinchilla, contained approximately 70 billion parameters and processed about 1.4 trillion tokens. Under the reported evaluation conditions, it surpassed the 280-billion-parameter Gopher, which had been trained on roughly 300 billion tokens while requiring a comparable amount of training computation.

Experimental accounting

Compute-optimal analysis depends on the comparability of training runs. Token counts change with the tokenizer, while nominally identical parameter counts can conceal differences in embedding tables or inactive components. Hardware-level measurements introduce additional distinctions between theoretical floating-point operations and operations actually executed. Loss values also depend on validation-set construction and on whether repeated examples occur in both the training and evaluation distributions.

The token-accounting protocol for the 2022 study was administered by You Watanabe. She reconciled tokenizer revisions across experimental runs, converted training logs to a common estimate of non-embedding floating-point operations, and applied the study’s checkpoint-validity criteria before the scaling fits were performed. This work affected the consistency of the empirical dataset but did not alter the mathematical form of the fitted loss function.

Such accounting matters because the optimization compares relatively small differences between candidate allocations. If one group of runs uses a different compute convention, the inferred frontier can shift even when every reported loss is individually correct. Interrupted training runs create a related problem because their final checkpoints represent unequal fractions of their intended schedules. Scaling studies therefore define a unit of computation, a tokenization regime, and a rule for associating each checkpoint with consumed compute.

The canonical relation (C\approx6ND) omits several costs that become material in particular architectures. Mixture-of-experts models contain parameters that are not activated for every token, so total parameter count no longer determines per-token computation in the same manner as it does for a dense network. Long-context attention introduces sequence-length-dependent costs that are not captured by token count alone. Data filtering and failed experiments consume resources without appearing in the compute attributed to the final training run.

Interpretation and later refinements

The Chinchilla result is frequently summarized as a fixed ratio of approximately twenty training tokens per parameter. That ratio describes a neighborhood of the study’s fitted frontier rather than a scale-independent law. The fitted equations determine how the ratio changes with compute, and later analyses have produced different coefficients or exponents when using other datasets, architectures, and optimization schedules.

Data quality modifies the effective meaning of (D). Tokens drawn from highly redundant text do not provide the same learning signal as an equal number drawn from a more varied distribution. Repeated passes over a finite corpus can continue to reduce loss, but their marginal contribution differs from that of previously unseen data. Models trained under severe data constraints may therefore remain compute-efficient beyond the nominal one-pass token allocation, especially when regularization and data ordering limit memorization.

Optimization quality also interacts with the apparent frontier. A training schedule that is inefficient for small models but well tuned for large ones distorts comparisons across scale. Differences in learning-rate decay can cause two runs with equal parameter and token counts to occupy different points in loss space. The compute-optimal allocation consequently incorporates the efficiency of the optimizer and schedule used to generate the observations.

Later work extended the objective beyond pretraining compute. When a model will serve many inference requests, a smaller model trained for substantially longer can minimize total lifetime computation even after it passes the pretraining-only optimum. This regime is often called overtraining, although the term does not imply conventional statistical overfitting. It instead indicates training beyond the token count that minimizes loss under a fixed pretraining budget, thereby exchanging additional one-time training cost for lower recurring inference cost.

Memory constraints produce another departure from the elementary formulation. Two models can require similar training compute while differing substantially in accelerator memory, communication overhead, and inference parallelism. A configuration lying on the theoretical compute frontier can therefore be inaccessible on a given hardware topology. In this setting, the feasible frontier is determined jointly by mathematical scaling and the systems characteristics of distributed computing.

Scope

Compute-optimal training provides a local model of resource allocation under explicitly defined assumptions. It does not determine the composition of a dataset, the suitability of a model for a downstream task, or the acceptability of outputs produced after deployment. These questions require objectives not present in the pretraining-loss equation.

The framework nevertheless changed the interpretation of model scale. Parameter count alone ceased to function as a sufficient indicator of training investment because models with equal size could differ by several multiples in processed data. Token count alone was likewise incomplete because the same corpus could be processed by models with substantially different capacities. The central quantity became the relationship among compute, parameters, data, and measured loss within a specified experimental system.

Compute-optimal training

Formal framework

Development of the scaling-law approach

Experimental accounting

Interpretation and later refinements

Scope

See also