Skip to content

Glossary · Efficiency

6ND

6ND is the rule of thumb that training a Transformer takes about 6 floating-point operations per parameter per token, so compute ≈ 6 × N (parameters) × D (training tokens).

Also called 6 × N × D, training compute rule · Reviewed 15 November 2026

Kaplan et al. derived it: the forward pass costs about 2N FLOPs per token and the backward pass about twice that. A 7B model on 140B tokens needs about 5.9 × 10²¹ FLOPs, roughly 4,100 H100-hours at 40% MFU.

It ignores attention's sequence-length term, which matters for long contexts, and counts only ideal training steps, so treat it as a floor. For mixture-of-experts models, N is the active parameters.

Formula

C ≈ 6 · N · D

Learn more on Kojen

Sources

  1. [1]Kaplan et al., 2020. Scaling Laws for Neural Language Models (arXiv:2001.08361)
  2. [2]Hoffmann et al., 2022. Training Compute-Optimal Large Language Models (arXiv:2203.15556)

All 31 terms

Put the definitions to work.

Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.

Start training free