Chinchilla: The 70B Model That Proved Bigger Is Not Always Better

For two years after OpenAI published its scaling laws paper in January 2020, the dominant strategy in large language model development was to scale parameters aggressively while keeping training data relatively fixed. The Kaplan et al. paper (arXiv:2001.08361) had concluded that larger models are significantly more sample-efficient and that compute-optimal training meant using very large models on modest data, stopping well before convergence. GPT-3 (175B parameters, 300B tokens), Gopher from DeepMind (280B parameters, 300B tokens), and Megatron-Turing NLG (530B parameters, 270B tokens) all followed this template. Then in March 2022, a team of 22 researchers at Google DeepMind published 'Training Compute-Optimal Large Language Models' (arXiv:2203.15556) and showed that the entire field had been training its models wrong.

The core finding: model size and tokens must scale together

The paper, led by Jordan Hoffmann and Sebastian Borgeaud, trained over 400 language models ranging from 70 million to over 16 billion parameters on between 5 billion and 500 billion tokens. Using three independent estimation methods, all three converged on the same result: for a fixed compute budget, the optimal strategy is to scale model size and the number of training tokens equally. If you double the model parameters, you should also double the training tokens. The Kaplan recommendation to prioritize parameters over data was wrong, or at least incomplete. The 2020 paper had held model quality fixed and asked how to minimize compute. The 2022 paper fixed compute and asked how to maximize quality. That shift in framing changed the answer entirely.

The three methods the paper used to arrive at this conclusion were: (1) training a series of models at fixed compute budgets while varying the size-to-token ratio, finding the best-performing allocation at each budget; (2) constructing IsoFLOP curves by fixing total floating-point operations and tracing how loss changes as size and tokens trade off against each other; and (3) fitting a parametric loss model to all 400-plus training runs and directly predicting the optimal allocation formula. All three methods produced consistent recommendations. The rule of thumb that emerged was that for compute-optimal training, each model parameter should see roughly 20 training tokens.

Chinchilla versus Gopher: same budget, very different results

To test the prediction, the authors trained Chinchilla using the same compute budget as Gopher but with the optimal allocation the new laws prescribed. Gopher used 280 billion parameters and 300 billion training tokens. Chinchilla used 70 billion parameters and 1.4 trillion tokens, four times as many. The parameter count dropped by 75 percent, and the token count quadrupled, keeping total compute roughly constant. Chinchilla uniformly outperformed Gopher (280B), GPT-3 (175B), Jurassic-1 from AI21 Labs (178B), and Megatron-Turing NLG (530B) across a broad range of downstream evaluation tasks. On MMLU (Measuring Massive Multitask Language Understanding), Chinchilla reached 67.5 percent average accuracy, more than 7 percentage points above Gopher's result. A 70B model trained on more data decisively beat a 530B model trained the old way.

Why the earlier laws missed this

The Kaplan paper's conclusion that larger models are more sample-efficient was technically accurate in a narrow sense: for a fixed dataset size, adding parameters helps more than adding tokens up to a point. But that framing treated dataset size as a constraint rather than a variable. In practice, compute budgets are the actual constraint for most training runs, not dataset size. When the question is reframed as how to allocate a fixed compute budget across model size and data simultaneously, the answer flips. Data becomes far more valuable than Kaplan's analysis suggested, because the prior analysis had never explored the regime of smaller models trained on much larger datasets. The 400-plus runs in the Chinchilla study explicitly explored that regime and found that models in the 70B-to-100B range trained on one to two trillion tokens were consistently beating models many times larger trained on less data.

Impact on the models that followed

The Chinchilla paper restructured how major labs planned training runs almost immediately. The most direct evidence is the LLaMA series from Meta AI, released in February 2023. The LLaMA-1 paper explicitly cited Chinchilla scaling laws as its design rationale. LLaMA-1 trained a 7B parameter model on 1 trillion tokens and a 65B model on 1.4 trillion tokens, matching the Chinchilla prescription closely. The goal was to produce models that were inference-efficient: smaller parameter counts mean faster inference at the same quality level, because active compute per token scales with parameters, not training tokens. A Chinchilla-optimal model is cheaper to serve than an undertrained larger model with comparable benchmark performance. LLaMA-2 pushed this further: the 7B and 13B variants trained on 2 trillion tokens, exceeding even the Chinchilla-optimal amount on the grounds that inference cost savings justify extra training.

• LLaMA-1 7B: trained on 1T tokens (Chinchilla-optimal for roughly 50B params, slightly overtrained by parameter count but cheaper to serve).
• LLaMA-1 65B: trained on 1.4T tokens, matching Chinchilla's formula for a model of that size.
• LLaMA-2 7B and 13B: trained on 2T tokens, deliberately over-training relative to compute-optimal to reduce inference cost.
• Mistral 7B (September 2023): used sliding window attention and trained on a large token dataset, inheriting the Chinchilla lesson about data volume.
• Google's PaLM 2 (2023): reported training at a smaller scale than PaLM-1 with more tokens, consistent with the compute-optimal shift.

The inference-cost argument and its limits

One implication of the Chinchilla result that the paper itself did not emphasize but the community quickly recognized is that compute-optimal training is not always deployment-optimal. If you are going to run inference on a model millions of times, it can be worth spending extra compute during training to produce a smaller model that achieves the same quality. A 7B model running at Chinchilla-optimal quality costs a fraction of a 30B undertrained model at the same benchmark score. This logic, sometimes called inference-optimal or over-training relative to Chinchilla, drove the trend toward heavily training smaller models that characterizes the 2023 to 2025 generation of open-weight releases. The key caveat is that Chinchilla's laws were derived for a specific loss function (cross-entropy on the Pile-like data distributions) and may not generalize perfectly to every architecture variant, tokenizer, or data mixture. Subsequent work has found that the optimal token-to-parameter ratio can shift depending on these factors, and that real-world constraints around data quality and availability matter as much as the formula. PromptingIndex covers Chinchilla alongside QLoRA, ZeRO, Mamba, and the Switch Transformer as part of its series on the architectural and training decisions that define how modern language models are built.