The Economics of Model Training
Lesson 3 of 4Intermediate 11 min

What training costs at each model size

From a $5 toy model to a nine-figure frontier run: estimates by size, checked against what real projects reported.

In this lesson you'll

  • Estimate the cost of a compute-optimal run at each model size
  • Read published training costs critically
  • Separate the cost of a final run from the cost of a whole project

H100 pricing varies widely. As of November 2026, the cheapest secure on-demand listing was about $2.90 an hour [1], Runpod charged $3.99 [2], Lambda's 8-GPU rate was set to become $4.59 on 16 November [3], and an AWS Capacity Block worked out at $5.97 [4]. The estimates below use $4 an hour as a round planning rate.

Estimates by model size

This table applies the 6ND rule at 20 tokens per parameter, with 40% MFU. These are our estimates of the final run only, so they're floors, not quotes.

ModelTokensH100-hoursCost at $4/hrTime on 8 H100s
125M2.5Babout 1about $5minutes
1B20B84about $340about 10 hours
7B140B4,100about $17,000about 3 weeks
13B260B14,000about $57,000about 2.5 months
70B1.4T413,000about $1.7Mabout 6 years

Cost grows with the square of model size when tokens scale alongside it, so a 10× larger model costs about 100× more. The last column shows why large runs need clusters: a 70B run that would take six years on one node takes about nine months on 64 GPUs, if it scales perfectly.

What real projects reported

RunSize and dataHardwareReported cost
nanochat [5]GPT-2 capability8 H100s, about 2 hoursabout $48
llm.c GPT-2 [6]1.6B, 33.6B tokens8 H100s, 24 hours$672
MPT-7B [7]6.7B, 1T tokens440 A100s, 9.5 daysabout $200,000
SmolLM2 [8]1.7B, 11T tokens256 H100sabout $250,000
DeepSeek-V3 [9]671B total, 37B active, 14.8T tokens2.788M H800-hours$5.576M at $2/hr

Meta didn't publish a dollar figure for Llama 3.1, but at $4 an hour its reported GPU-hours would come to about $5.8M for the 8B model and about $123M for the 405B [10]. Notice how much tokens matter. SmolLM2 is smaller than MPT-7B but cost more, because it trained on 11 times the data.

Reading a headline number

Published costs usually cover one thing: the final run. DeepSeek says this plainly. Its $5.576M assumes a $2 rental rate and excludes prior research and ablation experiments [9]. The model is also a mixture of experts with 37B of its 671B parameters active per token, so its compute is far below that of a dense 671B model.

A full project costs more. Across GPT-3, OPT-175B, GPT-4 and Gemini Ultra, Epoch AI estimated development costs at 47–64% hardware, 29–49% R&D staff including equity, and only 2–6% energy. It also found that the cost of the most compute-intensive runs had grown about 2.4× a year since 2016, and projected that the largest runs would pass $1 billion by 2027 [11].

In Kojen

Most teams don't need any row in that first table. The Forge runs prompt tuning and model search for real on hosted models, so you need no GPUs of your own. LoRA/QLoRA, full fine-tuning and RLHF for text models are previews with simulated metrics, so no text model weights are trained yet.

Key takeaways

  • With tokens scaled to match, cost grows with the square of model size.
  • GPT-2-class training now costs tens of dollars, 7B-class runs cost tens of thousands, and frontier runs cost nine figures.
  • Published figures usually cover the final run and leave out research, failed experiments and staff.

Practise in the dojo

Pick one project and write down the smallest model that could plausibly do its job. Estimate what training that size from scratch would cost, then compare it with running model search on the same task.

Open Projects

Check your understanding

Score 70 or more to complete the lesson. Progress is saved in this browser.

1. With tokens scaled at a fixed ratio to parameters, how much more does a 10× larger model cost to train?
2. What does DeepSeek-V3's $5.576M figure leave out?
3. In Epoch AI's breakdown of frontier development costs, which item was smallest?

Further reading

Sources

Checked against the original sources in November 2026.

  1. [1]GPUPerHour (2026). Cloud GPU pricing comparison (November 2026). gpuperhour.com.
  2. [2]Runpod (2026). GPU pricing (updated 4 November 2026). runpod.io.
  3. [3]Lambda (2026). AI cloud pricing (rates effective 16 November 2026). lambda.com.
  4. [4]Amazon Web Services (2026). Amazon EC2 Capacity Blocks for ML pricing. aws.amazon.com.
  5. [5]Karpathy, A. (2026). nanochat. GitHub · karpathy/nanochat.
  6. [6]Karpathy, A. (2024). Reproducing GPT-2 (1.6B) on one 8XH100 node in 24 hours for $672, in llm.c. GitHub · karpathy/llm.c discussion 677.
  7. [7]MosaicML (2023). Introducing MPT-7B: A New Standard for Open-Source, Commercially Usable LLMs. Databricks blog.
  8. [8]Allal, L. B. et al. (2025). SmolLM2: When Smol Goes Big — Data-Centric Training of a Small Language Model. arXiv:2502.02737.
  9. [9]DeepSeek-AI (2024). DeepSeek-V3 Technical Report. arXiv:2412.19437.
  10. [10]Meta (2024). Llama 3.1 model card. GitHub · meta-llama/llama-models.
  11. [11]Cottier, B. et al. (2024). The rising costs of training frontier AI models. arXiv:2405.21015.