How much compute a training run takes

A Transformer costs about 6 floating-point operations per parameter per training token ^[1]. That gives the most useful rule of thumb in training economics, where N is parameters and D is training tokens: compute ≈ 6 × N × D. Double the model or double the data, and you double the bill. Double both, and it quadruples.

How many tokens should a model see?

DeepMind's Chinchilla work found that for a fixed compute budget, model size and training tokens should grow together: each doubling of model size should come with a doubling of tokens ^[2]. Their 70B-parameter Chinchilla trained on 1.4 trillion tokens, which is where the popular shorthand of about 20 tokens per parameter comes from.

That ratio minimises training cost for a given quality, not lifetime cost. A smaller model is cheaper on every request it ever serves, so teams expecting heavy usage should train smaller models for longer than Chinchilla suggests ^[3]. Llama 3.1 8B saw about 15 trillion tokens, close to 1,900 tokens per parameter ^[4].

From FLOPs to GPU-hours

No GPU sustains its advertised peak. Model FLOPs utilisation (MFU) is the share of peak a run actually achieves, and Meta reported 38–43% for Llama 3 405B on up to 16,000 H100s ^[5]. An H100's dense BF16 peak is about 989 TFLOPS, half the 1,979 NVIDIA lists with sparsity ^[6], so 40% MFU gives roughly 396 TFLOPS of useful work.

Example: a 7B model on 140B tokens (20 tokens per parameter)

compute    = 6 × 7e9 × 140e9       = 5.9e21 FLOPs
throughput = 989 TFLOPS × 40% MFU  = 396 TFLOPS per H100
time       = 5.9e21 / 396e12       = 1.5e7 GPU-seconds
           = about 4,100 H100-hours
cost       = 4,100 × $4 per hour   = about $16,500

Treat the result as a floor

The formula counts only ideal training steps. Comparing it with the GPU-hours Meta published for Llama 3.1 ^[4] shows the gap, assuming 15 trillion tokens and 40% MFU: