Chinchilla Scaling Laws: The 2022 Paper That Showed GPT-3 Was Undertrained

The Chinchilla paper changed how large language models are trained. Published March 29, 2022 (arXiv:2203.15556) by Jordan Hoffmann and 21 colleagues at DeepMind, 'Training Compute-Optimal Large Language Models' ran a controlled experiment across more than 400 models and found that the models making headlines at the time, including GPT-3 and Gopher, were substantially undertrained. The finding was not a minor refinement. It overturned the prevailing principle that guided how compute budgets were allocated, and it produced a 70-billion-parameter model that outperformed a 280-billion-parameter one built on the same hardware budget.

The Kaplan laws and why they steered labs toward bigger models

In January 2020, Jared Kaplan and colleagues at OpenAI published 'Scaling Laws for Neural Language Models' (arXiv:2001.08361), which established that language model loss decreases as a power law with respect to model size, dataset size, and compute. The paper's central finding was that larger models are significantly more sample-efficient: for a fixed compute budget, you get more performance per FLOP by making the model bigger rather than by training it longer. The implication labs took from this was clear. Scale the model. Keep training data roughly constant. GPT-3 (175 billion parameters) was trained on approximately 300 billion tokens. Gopher (280 billion parameters, also from DeepMind, December 2021) was similarly trained on approximately 300 billion tokens. Both followed the Kaplan logic of big model, modest data.

How Hoffmann et al. designed the 2022 experiment

The Hoffmann et al. paper set out to pin down the compute-optimal frontier more precisely. The team trained over 400 language models ranging from 70 million to over 16 billion parameters on token budgets from 5 billion to 500 billion tokens. They applied three independent methods to estimate the optimal parameter-to-token ratio at each compute budget: fitting a parametric loss function over the full dataset of training runs, constructing a series of IsoFLOP profiles where compute is held constant and the model-to-token split is varied, and fitting a second parametric form to the fitted per-run optimal values. All three methods converged on the same answer: model size and training token count should scale at roughly equal rates. For every doubling of model size, the number of training tokens should also double.

This directly contradicted the Kaplan recommendation. Hoffmann et al. identified the difference as a methodological artifact in the earlier work: the 2020 paper had not trained models long enough to observe where the per-token data contribution flattens. When the experiment was designed to hold total compute constant and vary the model-to-token split, the optimal point was linear in both quantities, not biased toward model size.

What Chinchilla proved in practice

The team trained a model called Chinchilla as a direct empirical test. Chinchilla used the same compute budget as Gopher but reallocated it: 70 billion parameters trained on 1.4 trillion tokens, four times the data and one quarter the model size. The results were comprehensive. Chinchilla outperformed Gopher (280B parameters), GPT-3 (175B), Jurassic-1 (178B from AI21 Labs), and Megatron-Turing NLG (530B from Microsoft and NVIDIA) across a wide range of downstream evaluations.

• MMLU (a 57-subject benchmark covering STEM, humanities, and professional domains): Chinchilla reached 67.5 percent accuracy, more than 7 percentage points above Gopher.
• BIG-bench (a held-out suite of novel tasks assembled to resist benchmark contamination): Chinchilla set state-of-the-art at the time of publication.
• Reading comprehension, common-sense reasoning, and closed-book question answering: Chinchilla outperformed all comparison models in aggregate.

The practical implication went beyond benchmark rankings. Smaller models cost less to serve. A 70-billion-parameter model uses roughly one quarter the memory and inference compute of a 280-billion-parameter model. Chinchilla demonstrated that a well-trained smaller model can beat a poorly-trained larger one, so the inference savings came for free alongside the quality improvement.

The 20 tokens per parameter rule of thumb

The Chinchilla paper's findings reduce to a single approximation that practitioners reached for immediately: roughly 20 training tokens per model parameter is the compute-optimal ratio for a single training run, when inference cost is not factored in. For Chinchilla: 70 billion parameters multiplied by 20 gives 1.4 trillion tokens, which matches the actual training setup exactly. For a 7-billion-parameter model under the same principle, the target would be approximately 140 billion tokens.

This approximation is useful as a first-order guide but has two important limits. It does not say anything about what happens after deployment. And it assumes the practitioner has enough data available to hit the target token count without repeating samples. For internet-scale pretraining, data availability was not a binding constraint in 2022, but that assumption has grown less reliable as labs push toward tens of trillions of tokens.

How the field moved beyond the original recommendation

The Chinchilla paper reshaped how labs structured training runs. Meta's Llama (February 2023, arXiv:2302.13971, by Hugo Touvron and colleagues) explicitly cited the lower inference cost of smaller but longer-trained models as a core design motivation. The Llama 7-billion-parameter model was trained on approximately 1 trillion tokens and the 65-billion-parameter model on 1.4 trillion, both informed by the Chinchilla findings. Subsequent iterations pushed further: Llama 2 (2023) used 2 trillion training tokens, and Llama 3 (2024) used 15 trillion tokens, roughly ten times the Chinchilla-optimal amount for its parameter count.

The logic behind training far beyond Chinchilla-optimal is captured in a 2024 paper titled 'Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws' (arXiv:2401.00448) by Sardana, Portes, Doubov, and Frankle at MosaicML. Once you account for inference costs over a model's deployment lifetime, the compute-optimal training point shifts substantially. For a model expecting significant inference demand, it is cheaper overall to train a smaller model for much longer, reducing per-token serving costs, even if the training run itself consumes more compute than Chinchilla would prescribe. The paper found that for approximately 1 billion inference requests, a smaller model trained well beyond Chinchilla-optimal can reduce total lifetime compute by roughly 17 percent compared to a Chinchilla-optimal model of the same quality.

The Chinchilla finding stands on its own terms: model size and training tokens should scale together, and the dominant models of 2021 were undertrained. But the industry has since moved the target, trading even more training compute for smaller and cheaper-to-serve models. PromptingIndex covers the model layer and the systems layer together, because decisions made at training time determine what you can practically build at inference time.