Chinchilla scaling
Chinchilla scaling is DeepMind's finding that, for a fixed compute budget, model size and training tokens should grow in equal proportion, which works out to about 20 tokens per parameter.
Hoffmann et al. trained over 400 models and found earlier large models were undertrained. Their 70B Chinchilla, trained on 1.4T tokens, beat the 280B Gopher with the same compute.
Compute-optimal minimises training cost for a given quality, not lifetime cost. Models that will serve heavy traffic are usually trained far longer: Llama 3.1 8B saw about 15T tokens, nearly 1,900 per parameter.
Formula
D_opt ≈ 20 · N
Learn more on Kojen
- Academy: How much compute a training run takes
- Blog: What it really costs to train an LLM
- Reference: fine-tuning and training costs
Related terms
- 6ND6ND is the rule of thumb that training a Transformer takes about 6 floating-point operations per parameter per token, so compute ≈ 6 × N (parameters) × D (training tokens)..
- MFUModel FLOPs utilisation (MFU) is the share of a GPU's theoretical peak throughput that a training run spends on the model's own arithmetic.
Sources
- Hoffmann et al., 2022. Training Compute-Optimal Large Language Models (arXiv:2203.15556)
Put the definitions to work.
Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.
Start training free