What it costs to train a 7B model from scratch
A compute-optimal run for a 7B-parameter model, 140B tokens at 20 per parameter, takes about 4,129 H100-hours, around $18,125 at $4.39 an hour. Training on 2,000B tokens (2T tokens, the Llama 2 budget) raises that to about $258,932. Tens of thousands of dollars compute-optimal and needs a multi-GPU node just to hold the training state; production-quality 7B runs cost hundreds of thousands.
Compute-optimal: 140B tokens
| GPU | GPU-hours | Cost | Time on 8 | Time on 64 | Time on 512 | GPUs to hold state |
|---|---|---|---|---|---|---|
| RTX 5090209.5 TFLOPS · $1.39/hr | 19,491 | $27,092 | 3.3 months | 13 days | 38 hours | 4 |
| A100 80GB312 TFLOPS · $2.19/hr | 13,088 | $28,662 | 2.2 months | 9 days | 26 hours | 2 |
| H100 SXM989 TFLOPS · $4.39/hr | 4,129 | $18,125 | 22 days | 3 days | 8.1 hours | 2 |
| H200989 TFLOPS · $5.29/hr | 4,129 | $21,841 | 22 days | 3 days | 8.1 hours | 1 |
| B2002,250 TFLOPS · $7.99/hr | 1,815 | $14,500 | 9 days | 28 hours | 3.5 hours | 1 |
Longer training: 2,000B tokens
| GPU | GPU-hours | Cost | Time on 8 | Time on 64 | Time on 512 | GPUs to hold state |
|---|---|---|---|---|---|---|
| RTX 5090209.5 TFLOPS · $1.39/hr | 278,441 | $387,033 | 4.0 years | 6.0 months | 23 days | 4 |
| A100 80GB312 TFLOPS · $2.19/hr | 186,966 | $409,455 | 2.7 years | 4.0 months | 15 days | 2 |
| H100 SXM989 TFLOPS · $4.39/hr | 58,982 | $258,932 | 10.1 months | 38 days | 5 days | 2 |
| H200989 TFLOPS · $5.29/hr | 58,982 | $312,016 | 10.1 months | 38 days | 5 days | 1 |
| B2002,250 TFLOPS · $7.99/hr | 25,926 | $207,148 | 4.4 months | 17 days | 2 days | 1 |
Compute = 6 × 7e9 parameters × tokens, at 40% utilisation of dense BF16 peak, with perfect scaling across GPUs. Memory for weights, gradients and Adam state is 112 GB before activations.
What real runs of this size reported
| Run | Size and data | Reported |
|---|---|---|
| Llama 2 7B | 2T tokens | 184,320 A100-hours |
| MPT-7B | 6.7B parameters, 1T tokens, 440 A100s for 9.5 days | about $200,000 |
Published GPU-hours for Llama 3.1 came in 1.2–2.9× above the 6ND estimate; budget for that gap, plus experiments, failed runs and data work, before you treat these numbers as a quote.
Change the assumptions
Pre-filled for 7B parameters at 140B tokens on H100s.
$18,125
4,129
22 days
5.9e21 FLOPs
Full training needs about 112 GB for weights, gradients and optimiser state before activations, so at least 2 H100 SXMs just to hold the model. Fine-tuning with QLoRA instead needs about 4 GB for 4-bit weights, which fits on a single 32 GB card.
Frequently asked questions
- How much does it cost to train a 7B model?
- About $18,125 on H100s at $4.39/hour for a compute-optimal run of 140B tokens (6 × parameters × tokens at 40% utilisation). At 2,000B tokens (2T tokens, the Llama 2 budget) it is about $258,932. Treat both as floors: published runs used 1.2–2.9× more GPU-hours than the formula.
- How long does it take to train a 7B model?
- About 22 days on one 8-GPU H100 node for 140B tokens, assuming perfect scaling, or 3 days on 64 GPUs.
- How much GPU memory does training a 7B model need?
- About 112 GB for weights, gradients and Adam state (16 bytes per parameter) before activations, so at least 2 × 80 GB GPUs with the state sharded.
- Which GPU is cheapest for training a 7B model?
- At on-demand November 2026 rates, the B200 ($7.99/hour) gives the lowest compute cost, about $14,500, by our estimate. Peak FLOPS aren't delivered performance, so benchmark your own code before a large run.
Fine-tuning instead
Open models of this size already carry far more pretraining than most budgets allow. Adapting one costs a tiny fraction:
- Fine-tune Mistral 7B: cost$0.61 per 10M tokens
- Fine-tune Qwen2.5 7B: cost$0.64 per 10M tokens
- Fine-tune Llama 3.1 8B: cost$0.67 per 10M tokens
- Train a 125M modelFrom scratch
- Train a 1B modelFrom scratch
- Train a 3B modelFrom scratch
- Train a 13B modelFrom scratch
- Train a 70B modelFrom scratch
The method in depth is in the free Academy course The Economics of Model Training and the article What it really costs to train an LLM; terms are defined in the glossary under 6ND, Chinchilla scaling and MFU.
Sources
- Kaplan et al., 2020. Scaling Laws for Neural Language Models (arXiv:2001.08361)
- Hoffmann et al., 2022. Training Compute-Optimal Large Language Models (arXiv:2203.15556)
- Meta, 2024. The Llama 3 Herd of Models (arXiv:2407.21783)
- Rajbhandari et al., 2019. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (arXiv:1910.02054)
- Runpod. GPU pricing (updated 12 November 2026)
- Touvron et al., 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models (arXiv:2307.09288)
- MosaicML. Introducing MPT-7B
You probably don't need to pretrain.
Kojen's prompt tuning and model search run on hosted models against your own test cases, and text LoRA trains on real GPUs (for accounts with GPU training enabled) when you need weights to change.
Start training free