6ND
6ND is the rule of thumb that training a Transformer takes about 6 floating-point operations per parameter per token, so compute ≈ 6 × N (parameters) × D (training tokens).
Kaplan et al. derived it: the forward pass costs about 2N FLOPs per token and the backward pass about twice that. A 7B model on 140B tokens needs about 5.9 × 10²¹ FLOPs, roughly 4,100 H100-hours at 40% MFU.
It ignores attention's sequence-length term, which matters for long contexts, and counts only ideal training steps, so treat it as a floor. For mixture-of-experts models, N is the active parameters.
Formula
C ≈ 6 · N · D
Learn more on Kojen
- Academy: How much compute a training run takes
- Tool: LLM training cost calculator
- Blog: What it really costs to train an LLM
Related terms
- MFUModel FLOPs utilisation (MFU) is the share of a GPU's theoretical peak throughput that a training run spends on the model's own arithmetic.
- Chinchilla scalingChinchilla scaling is DeepMind's finding that, for a fixed compute budget, model size and training tokens should grow in equal proportion, which works out to about 20 tokens per parameter..
- MoEA mixture-of-experts (MoE) model replaces each dense feed-forward layer with many expert networks and a router that sends each token to a few of them, so only a fraction of the parameters work on any token..
Sources
- Kaplan et al., 2020. Scaling Laws for Neural Language Models (arXiv:2001.08361)
- Hoffmann et al., 2022. Training Compute-Optimal Large Language Models (arXiv:2203.15556)
Put the definitions to work.
Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.
Start training free