Academy
Course 250Intermediate

The Economics of Model Training

Estimate a training run before you pay for one. Learn the arithmetic behind compute budgets, why memory decides which hardware you can use, what models of each size really cost, and when to rent, buy or skip training altogether. Prices are as of November 2026.

Stacks of obsidian coins with glowing red seams rising from left to right, the tallest topped with a red glass cube

Lessons

  1. 1. How much compute a training run takes

    Two numbers, model size and token count, give you a first estimate of any run. Here's the arithmetic, and where it falls short.

    10 min

  2. 2. Memory first: what hardware can train what

    Speed decides how long a run takes. Memory decides whether it runs at all, and training needs far more of it than running the same model.

    10 min

  3. 3. What training costs at each model size

    From a $5 toy model to a nine-figure frontier run: estimates by size, checked against what real projects reported.

    11 min

  4. 4. Rent, buy or don't train at all

    Most teams should rent, and many shouldn't train at all. Here's how to work out which applies to you.

    9 min

Sources

Every source cited across this course's lessons.

  1. [1]Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361.
  2. [2]Hoffmann, J. et al. (2022). Training Compute-Optimal Large Language Models. NeurIPS 2022 · arXiv:2203.15556.
  3. [3]Sardana, N. et al. (2023). Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws. ICML 2024 · arXiv:2401.00448.
  4. [4]Meta (2024). Llama 3.1 model card. GitHub · meta-llama/llama-models.
  5. [5]Llama Team, AI @ Meta (2024). The Llama 3 Herd of Models. arXiv:2407.21783.
  6. [6]NVIDIA (2026). NVIDIA H100 Tensor Core GPU. nvidia.com.
  7. [7]Rajbhandari, S. et al. (2019). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. SC 2020 · arXiv:1910.02054.
  8. [8]Hu, E. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022 · arXiv:2106.09685.
  9. [9]Dettmers, T. et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023 · arXiv:2305.14314.
  10. [10]NVIDIA (2025). NVIDIA RTX Blackwell GPU Architecture. NVIDIA whitepaper.
  11. [11]Lenovo Press (2026). NVIDIA HGX B200 product guide. lenovopress.lenovo.com.
  12. [12]Runpod (2026). GPU pricing (updated 4 November 2026). runpod.io.
  13. [13]GPUPerHour (2026). Cloud GPU pricing comparison (November 2026). gpuperhour.com.
  14. [14]Lambda (2026). AI cloud pricing (rates effective 16 November 2026). lambda.com.
  15. [15]Amazon Web Services (2026). Amazon EC2 Capacity Blocks for ML pricing. aws.amazon.com.
  16. [16]Karpathy, A. (2026). nanochat. GitHub · karpathy/nanochat.
  17. [17]Karpathy, A. (2024). Reproducing GPT-2 (1.6B) on one 8XH100 node in 24 hours for $672, in llm.c. GitHub · karpathy/llm.c discussion 677.
  18. [18]MosaicML (2023). Introducing MPT-7B: A New Standard for Open-Source, Commercially Usable LLMs. Databricks blog.
  19. [19]Allal, L. B. et al. (2025). SmolLM2: When Smol Goes Big — Data-Centric Training of a Small Language Model. arXiv:2502.02737.
  20. [20]DeepSeek-AI (2024). DeepSeek-V3 Technical Report. arXiv:2412.19437.
  21. [21]Cottier, B. et al. (2024). The rising costs of training frontier AI models. arXiv:2405.21015.
  22. [22]Tom's Hardware (2026). GPU price tracking 2026 (updated 3 November 2026). tomshardware.com.
  23. [23]U.S. Energy Information Administration (2026). Electricity Monthly Update: end-use prices (August 2026 data). eia.gov.