MFU
Model FLOPs utilisation (MFU) is the share of a GPU's theoretical peak throughput that a training run spends on the model's own arithmetic. It is the main efficiency number for turning FLOPs into GPU-hours.
The PaLM paper defined MFU as observed tokens per second times the model's FLOPs per token, divided by peak FLOPS; recomputation for gradient checkpointing doesn't count. PaLM reached 46% on TPU v4, and Meta reported 38–43% for Llama 3 405B on H100s.
Planning at about 40% is reasonable for a well-tuned large run; small or unoptimised jobs often get far less.
Formula
MFU = (tokens/s × FLOPs per token) / peak FLOPS
Learn more on Kojen
- Academy: How much compute a training run takes
- Tool: LLM training cost calculator
- Reference: fine-tuning and training costs
Related terms
- 6ND6ND is the rule of thumb that training a Transformer takes about 6 floating-point operations per parameter per token, so compute ≈ 6 × N (parameters) × D (training tokens)..
- Chinchilla scalingChinchilla scaling is DeepMind's finding that, for a fixed compute budget, model size and training tokens should grow in equal proportion, which works out to about 20 tokens per parameter..
- Mixed precisionMixed-precision training runs most arithmetic in 16-bit floating point (FP16 or BF16) while keeping a 32-bit master copy of the weights for updates, roughly halving memory traffic and using fast tensor cores..
Sources
- Chowdhery et al., 2022. PaLM: Scaling Language Modeling with Pathways (arXiv:2204.02311)
- Meta, 2024. The Llama 3 Herd of Models (arXiv:2407.21783)
Put the definitions to work.
Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.
Start training free