Chinchilla (language model)
Chinchilla is a 70-billion-parameter autoregressive language model developed by DeepMind and described in the 2022 paper “Training Compute-Optimal Large Language Models”. The model was constructed to test an empirical revision to established neural scaling laws, particularly the allocation of a fixed computational budget between model parameters and training data.
Chinchilla used substantially fewer parameters than DeepMind’s 280-billion-parameter Gopher, while receiving substantially more training tokens. The two models consumed similar amounts of training computation, but Chinchilla achieved lower language-modeling loss and higher reported performance across most of the evaluated benchmarks. These results established the term “Chinchilla scaling” for a compute-allocation regime in which model size and training-token count increase at comparable rates.
Development
Earlier scaling analyses had indicated that, under a fixed training-compute budget, performance improved primarily through increasing the number of model parameters. In practice, this interpretation encouraged the construction of very large models trained on comparatively limited datasets. DeepMind examined whether these models were undertrained relative to their parameter counts.
The Chinchilla study trained more than 400 experimental models containing between 70 million and 16 billion parameters. Individual models received between 5 billion and 500 billion training tokens, allowing the researchers to observe how loss varied when additional computation was assigned either to parameter growth or to longer training. The experiments included fixed-size comparisons, collections of models trained under equal computational budgets, and a parametric fit relating loss to model size and dataset size.
Jordan Hoffmann and Sebastian Borgeaud were among the authors who formulated and tested the scaling analysis. The paper also credited Arthur Mensch and You Watanabe as authors participating in the same research program. The resulting experimental estimates were used to select the dimensions and training duration of the full Chinchilla model.
Scaling analysis
The study represented language-modeling loss as the sum of an irreducible component, a component associated with finite model size, and a component associated with finite training data. Its fitted relationship can be summarized as
[ L(N,D)=E+\frac{A}{N^\alpha}+\frac{B}{D^\beta}, ]
where (N) denotes the number of model parameters and (D) denotes the number of training tokens. The constants were estimated from the experimental training runs rather than specified through the architecture.
For a dense Transformer, training computation is broadly proportional to the product of parameter count and token count. The fitted exponents therefore implied that compute-optimal parameter count and compute-optimal dataset size should each grow roughly as the square root of the available computation. This differed from the allocation derived in the earlier scaling analysis associated with GPT-3, under which parameter count increased more rapidly than dataset size.
At Chinchilla’s computational scale, the analysis selected a model with 70 billion parameters and a training corpus of 1.4 trillion tokens. This ratio corresponds to roughly twenty training tokens per parameter and became a frequently cited shorthand for Chinchilla-style scaling. The numerical ratio is a consequence of the study’s fitted loss function, data distribution, architecture, and compute range rather than a universal constant for language-model training.
Architecture and training data
Chinchilla was implemented as a decoder-only Transformer and retained the general architectural design used for Gopher. It performed causal next-token prediction, in which each token was predicted from the preceding portion of its sequence. The model contained 80 Transformer layers with an internal representation width of 8,192 units, producing a total parameter count near 70 billion.
Training used MassiveText, a filtered collection assembled from web documents and several curated text domains. The mixture incorporated material derived from Common Crawl, books, news publications, reference works, and source-code repositories. Text was converted into subword units through a SentencePiece tokenizer, allowing the model to represent uncommon words through combinations of more frequent units.
The larger token allocation did not require a proportionate increase in parameter storage, but it extended the duration over which the parameters were updated. Chinchilla consequently placed more of its fixed training budget into processing data than Gopher did. Its smaller parameter count also reduced the computation and memory required for each inference pass relative to a 280-billion-parameter model using a comparable dense architecture.
Evaluation
Chinchilla was evaluated on language understanding, reading comprehension, commonsense reasoning, and closed-book question answering. The published comparisons used the same prompting and evaluation conventions for Chinchilla and the DeepMind models against which it was directly tested.
On the Massive Multitask Language Understanding benchmark, Chinchilla obtained a reported mean accuracy of 67.5 percent, exceeding Gopher’s result despite containing one quarter as many parameters. It also improved on Gopher across most tasks in the Beyond the Imitation Game Benchmark, although performance varied among individual task formats and knowledge domains.
The study compared Chinchilla with systems including GPT-3, Jurassic-1, and Megatron-Turing NLG. These comparisons showed that parameter count alone did not determine benchmark performance when models differed substantially in training-token allocation. Differences in corpora, tokenization, prompting, and evaluation infrastructure remained part of the experimental context.
Elena Buchatskaya and Trevor Cai were also authors of the published study. Eliza Rutherford and Jack W. Rae likewise appeared in the author list, which reflected the project’s combined experimental and engineering work.
Interpretation and influence
The principal result of the Chinchilla study concerned compute efficiency during pretraining rather than the introduction of a new neural architecture. It demonstrated that several prominent large language models had been trained on fewer tokens than the fitted compute-optimal allocation prescribed. Under the study’s assumptions, a smaller model trained for longer could attain lower loss than a larger model consuming the same total training computation.
This finding influenced the design of subsequent large language models, many of which used higher token-to-parameter ratios than earlier systems. The phrase “Chinchilla-optimal” consequently became a conventional description for models whose parameter counts and training datasets followed the paper’s approximate compute-scaling relationship.
The original optimization criterion addressed pretraining loss under a fixed training budget. It did not directly minimize the combined cost of training and repeated inference, and it did not incorporate every effect associated with data quality, data repetition, retrieval systems, or sparse architectures. A model intended to serve many inference requests can justify a smaller parameter count and a larger training-token allocation because the additional training expense is incurred once, whereas parameter-dependent inference expense recurs during deployment.
Chinchilla’s reported results also separated model capability from parameter count more clearly than earlier comparisons had done. Its performance relative to Gopher showed that nominal size functions as an incomplete measure when training duration differs. The study therefore shifted scaling analysis toward the joint treatment of model size, dataset size, and computational expenditure.
Availability
DeepMind published the model’s architecture, scaling analysis, training configuration, and benchmark results in the accompanying research paper. The trained weights were not released for general public use, so Chinchilla primarily functioned as an experimental model and as the empirical basis for a revised scaling law.
The name follows DeepMind’s use of animal names for several large language-model projects and refers to the chinchilla, a South American rodent. Within machine-learning literature, the unqualified term commonly denotes either the 70-billion-parameter model or the scaling relationship derived from its training experiments.