How the estimate works
A Transformer takes about six floating-point operations per parameter per training token, so a run needs roughly 6 × N × D FLOPs. Real throughput is the GPU's dense BF16 peak times its utilisation, so dividing gives GPU-seconds, and the hourly rate turns those into dollars.
compute = 6 × 7e9 parameters × 140e9 tokens = 5.9e21 FLOPs throughput = 989 TFLOPS × 40% utilisation = 396 TFLOPS time = 5.9e21 / 396e12 = about 4,100 H100-hours cost = 4,100 hours × $4 = about $16,500
Because compute-optimal training grows tokens with parameters, cost rises with the square of model size: a model ten times larger costs about a hundred times more. The result covers the final run only, so treat it as a floor. Meta's published GPU-hours for Llama 3.1 came in well above this formula, and real budgets also pay for experiments, failures, data and people.
The full method, real published run costs and what headline numbers leave out are in What it really costs to train an LLM. To work through it step by step, take The Economics of Model Training, a free Academy course. To check whether a model fits in memory for fine-tuning, use the fine-tuning VRAM calculator.
Frequently asked questions
- How is LLM training cost calculated?
- A Transformer takes about 6 floating-point operations per parameter per training token, so compute is roughly 6 × parameters × tokens. Divide by real GPU throughput (peak FLOPS × utilisation) to get GPU-hours, then multiply by the hourly price.
- How many tokens should I train on?
- DeepMind's Chinchilla results put the compute-optimal ratio at about 20 tokens per parameter. Many recent small models train far longer, such as SmolLM2 1.7B on 11 trillion tokens, because a smaller model trained longer is cheaper to serve.
- What GPU utilisation should I assume?
- Meta reported 38–43% model FLOPs utilisation training Llama 3 405B on H100s. 40% is a reasonable planning number for a well-tuned run; small or unoptimised runs often get less.
- Is the calculator's number the full cost of building a model?
- No. It covers one final training run. Real budgets also pay for experiments and failed runs, hardware interruptions, data collection and cleaning, staff and serving. Treat the result as a floor.
- Should I fine-tune instead of pretraining?
- Almost always. Open models already contain millions of dollars of pretraining, and a QLoRA fine-tune of a 7B–13B model fits on a single 32 GB GPU and can take about a day.