Skip to content

Cost reference

What it costs to train a 70B model from scratch

A compute-optimal run for a 70B-parameter model, 1,400B tokens at 20 per parameter, takes about 412,875 H100-hours, around $1.8M at $4.39 an hour. Training on 15,000B tokens (15T tokens, the Llama 3.1 budget) raises that to about $19M. Millions of dollars and hundreds of GPUs. Fine-tune an existing 70B model instead unless you are building a foundation model.

By Will Schulz · Checked 15 November 2026 · Prices as of November 2026

Compute-optimal: 1,400B tokens

GPUGPU-hoursCostTime on 8Time on 64Time on 512GPUs to hold state
RTX 5090209.5 TFLOPS · $1.39/hr1.9M$2.7M27.8 years3.5 years5.2 months35
A100 80GB312 TFLOPS · $2.19/hr1.3M$2.9M18.7 years2.3 years3.5 months14
H100 SXM989 TFLOPS · $4.39/hr412,875$1.8M5.9 years8.8 months34 days14
H200989 TFLOPS · $5.29/hr412,875$2.2M5.9 years8.8 months34 days8
B2002,250 TFLOPS · $7.99/hr181,481$1.5M2.6 years3.9 months15 days7

Longer training: 15,000B tokens

GPUGPU-hoursCostTime on 8Time on 64Time on 512GPUs to hold state
RTX 5090209.5 TFLOPS · $1.39/hr20.9M$29M298.0 years37.2 years4.7 years35
A100 80GB312 TFLOPS · $2.19/hr14.0M$31M200.1 years25.0 years3.1 years14
H100 SXM989 TFLOPS · $4.39/hr4.4M$19M63.1 years7.9 years11.8 months14
H200989 TFLOPS · $5.29/hr4.4M$23M63.1 years7.9 years11.8 months8
B2002,250 TFLOPS · $7.99/hr1.9M$16M27.7 years3.5 years5.2 months7

Compute = 6 × 70e9 parameters × tokens, at 40% utilisation of dense BF16 peak, with perfect scaling across GPUs. Memory for weights, gradients and Adam state is 1,120 GB before activations.

What real runs of this size reported

RunSize and dataReported
Llama 2 70B2T tokens1,720,320 A100-hours
Llama 3.1 70Babout 15T tokens7.0M H100-hours

Published GPU-hours for Llama 3.1 came in 1.2–2.9× above the 6ND estimate; budget for that gap, plus experiments, failed runs and data work, before you treat these numbers as a quote.

Change the assumptions

Pre-filled for 70B parameters at 1,400B tokens on H100s.

Estimated cost

$1.8M

H100 SXM-hours

412,875

Time on 8 GPUs

5.9 years

Compute

5.9e23 FLOPs

Full training needs about 1,120 GB for weights, gradients and optimiser state before activations, so at least 14 H100 SXMs just to hold the model. Fine-tuning with QLoRA instead needs about 35 GB for 4-bit weights.

Frequently asked questions

How much does it cost to train a 70B model?
About $1.8M on H100s at $4.39/hour for a compute-optimal run of 1,400B tokens (6 × parameters × tokens at 40% utilisation). At 15,000B tokens (15T tokens, the Llama 3.1 budget) it is about $19M. Treat both as floors: published runs used 1.2–2.9× more GPU-hours than the formula.
How long does it take to train a 70B model?
About 5.9 years on one 8-GPU H100 node for 1,400B tokens, assuming perfect scaling, or 8.8 months on 64 GPUs.
How much GPU memory does training a 70B model need?
About 1,120 GB for weights, gradients and Adam state (16 bytes per parameter) before activations, so at least 14 × 80 GB GPUs with the state sharded.
Which GPU is cheapest for training a 70B model?
At on-demand November 2026 rates, the B200 ($7.99/hour) gives the lowest compute cost, about $1.5M, by our estimate. Peak FLOPS aren't delivered performance, so benchmark your own code before a large run.

Fine-tuning instead

Open models of this size already carry far more pretraining than most budgets allow. Adapting one costs a tiny fraction:

The method in depth is in the free Academy course The Economics of Model Training and the article What it really costs to train an LLM; terms are defined in the glossary under 6ND, Chinchilla scaling and MFU.

Sources

  1. [1]Kaplan et al., 2020. Scaling Laws for Neural Language Models (arXiv:2001.08361)
  2. [2]Hoffmann et al., 2022. Training Compute-Optimal Large Language Models (arXiv:2203.15556)
  3. [3]Meta, 2024. The Llama 3 Herd of Models (arXiv:2407.21783)
  4. [4]Rajbhandari et al., 2019. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (arXiv:1910.02054)
  5. [5]Runpod. GPU pricing (updated 12 November 2026)
  6. [6]Touvron et al., 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models (arXiv:2307.09288)
  7. [7]Meta. Llama 3.1 model card

You probably don't need to pretrain.

Kojen's prompt tuning and model search run on hosted models against your own test cases, and text LoRA trains on real GPUs (for accounts with GPU training enabled) when you need weights to change.

Start training free