Training a GPT-2-class model now costs about $48. At today's rental rates, the GPU-hours Meta reported for Llama 3.1 405B would cost over $100 million. The distance between them comes down to a short piece of arithmetic that anyone can run before spending a dollar.
Here's that arithmetic, what models of each size cost as of November 2026, and what the headline numbers leave out.
The formula behind every training budget
A Transformer takes about six floating-point operations per parameter per training token.[1] So total compute is roughly 6 × parameters × tokens.
How many tokens? DeepMind's Chinchilla paper found that model size and data should grow together, and its 70B model trained on 1.4 trillion tokens.[2] That ratio is the source of the common rule of about 20 tokens per parameter.
To turn FLOPs into time, you need real throughput, not peak. Meta reported 38–43% utilisation training Llama 3 405B.[3] An H100's dense BF16 peak is about 989 TFLOPS, half the 1,979 that NVIDIA lists with sparsity.[4] At 40%, that's about 396 TFLOPS. Here's a 7B model on 140 billion tokens:
compute = 6 × 7e9 parameters × 140e9 tokens = 5.9e21 FLOPs throughput = 989 TFLOPS × 40% utilisation = 396 TFLOPS time = 5.9e21 / 396e12 = about 4,100 H100-hours cost = 4,100 hours × $4 = about $16,500
We use $4 an hour because H100 rates vary so much. In early November 2026, Runpod charged $3.99,[5] the cheapest secure on-demand listing on one tracker was $2.90,[6] Lambda's 8-GPU rate was due to become $4.59 on 16 November,[7] and an AWS Capacity Block came to $5.97.[8]
What each model size costs
These are our estimates using that method at 20 tokens per parameter. They cover the final run only.
| Model | Tokens | H100-hours | Cost at $4/hr |
|---|---|---|---|
| 125M | 2.5B | about 1 | about $5 |
| 1B | 20B | 84 | about $340 |
| 7B | 140B | 4,100 | about $17,000 |
| 13B | 260B | 14,000 | about $57,000 |
| 70B | 1.4T | 413,000 | about $1.7M |
Cost rises with the square of model size, because parameters and tokens grow together. A model ten times larger costs about a hundred times more.
Treat these as floors. We compared the formula with the GPU-hours Meta published for Llama 3.1, assuming 15 trillion tokens.[9] The published figures came in 1.2× higher for the 405B model, 1.6× for the 70B and 2.9× for the 8B. The model card doesn't say why, but a budget should allow for it.
What real runs reported
| Run | Size and data | Hardware | Reported cost |
|---|---|---|---|
| nanochat[10] | GPT-2 capability | 8 H100s, about 2 hours | about $48 |
| llm.c GPT-2[11] | 1.6B, 33.6B tokens | 8 H100s, 24 hours | $672 |
| MPT-7B[12] | 6.7B, 1T tokens | 440 A100s, 9.5 days | about $200,000 |
| SmolLM2[13] | 1.7B, 11T tokens | 256 H100s | about $250,000 |
| DeepSeek-V3[14] | 671B total, 37B active, 14.8T tokens | 2.788M H800-hours | $5.576M at $2/hr |
Meta didn't publish a dollar figure for Llama 3.1. At $4 an hour, its reported GPU-hours would come to about $5.8M for the 8B model and about $123M for the 405B. Those are our calculations, not Meta's numbers.
Token count drives a lot of this. SmolLM2 has a quarter of MPT-7B's parameters but cost more to train, because it saw 11 times the data. That's a deliberate trade: a small model trained for longer is cheaper to run on every request afterwards.[15]
What the headline number leaves out
DeepSeek-V3's $5.576M became famous as the price of a frontier model. The report itself is more careful: the figure assumes $2 an hour and excludes prior research and ablation experiments.[14] It's the cost of the final run, not the cost of getting there.
Epoch AI looked at full development costs for GPT-3, OPT-175B, GPT-4 and Gemini Ultra. It estimated 47–64% went on hardware, 29–49% on R&D staff including equity, and only 2–6% on energy. It also found that the cost of the most compute-intensive runs had grown about 2.4× a year since 2016, and projected that the largest would pass $1 billion by 2027.[16]
A realistic budget also has to cover:
- Experiments before the final run, including the ones that fail.
- Hardware failures. Meta logged 419 unexpected interruptions in 54 days of Llama 3 training, about eight a day.[3]
- Data, which takes real work to collect, clean and license.
- Serving, which can outweigh training over a popular model's life.[15]
What this means for your team
Unless you're doing research, you probably shouldn't pretrain at all. Open models already contain millions of dollars of training you can start from. Fine-tuning one with QLoRA can take about a day on a single GPU,[17] and better prompts or a better-suited model may get you there with no training.
To work through these estimates yourself, take The Economics of Model Training, a free four-lesson Academy course.
