Skip to content

Glossary · Efficiency

Chinchilla scaling

Chinchilla scaling is DeepMind's finding that, for a fixed compute budget, model size and training tokens should grow in equal proportion, which works out to about 20 tokens per parameter.

Also called Chinchilla-optimal, compute-optimal training, 20 tokens per parameter · Reviewed 15 November 2026

Hoffmann et al. trained over 400 models and found earlier large models were undertrained. Their 70B Chinchilla, trained on 1.4T tokens, beat the 280B Gopher with the same compute.

Compute-optimal minimises training cost for a given quality, not lifetime cost. Models that will serve heavy traffic are usually trained far longer: Llama 3.1 8B saw about 15T tokens, nearly 1,900 per parameter.

Formula

D_opt ≈ 20 · N

Learn more on Kojen

Sources

  1. [1]Hoffmann et al., 2022. Training Compute-Optimal Large Language Models (arXiv:2203.15556)

All 31 terms

Put the definitions to work.

Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.

Start training free