Skip to content

Cost reference

What it costs to train a 125M model from scratch

A compute-optimal run for a 125M-parameter model, 2.5B tokens at 20 per parameter, takes about 1.3 H100-hours, around $5.78 at $4.39 an hour. Training on 10B tokens (80 tokens per parameter, typical of GPT-2-small reproductions) raises that to about $23.12. A weekend project: one rented GPU trains it in hours for a few dollars. It is a teaching model, not a useful assistant.

By Will Schulz · Checked 15 November 2026 · Prices as of November 2026

Compute-optimal: 2.5B tokens

GPUGPU-hoursCostTime on 8Time on 64Time on 512GPUs to hold state
RTX 5090209.5 TFLOPS · $1.39/hr6.2$8.6447 minutes6 minutes1 minutes1
A100 80GB312 TFLOPS · $2.19/hr4.2$9.1431 minutes4 minutes1 minutes1
H100 SXM989 TFLOPS · $4.39/hr1.3$5.7810 minutes1 minutes1 minutes1
H200989 TFLOPS · $5.29/hr1.3$6.9610 minutes1 minutes1 minutes1
B2002,250 TFLOPS · $7.99/hr0.58$4.624 minutes1 minutes1 minutes1

Longer training: 10B tokens

GPUGPU-hoursCostTime on 8Time on 64Time on 512GPUs to hold state
RTX 5090209.5 TFLOPS · $1.39/hr25$34.563.1 hours23 minutes3 minutes1
A100 80GB312 TFLOPS · $2.19/hr17$36.562.1 hours16 minutes2 minutes1
H100 SXM989 TFLOPS · $4.39/hr5.3$23.1239 minutes5 minutes1 minutes1
H200989 TFLOPS · $5.29/hr5.3$27.8639 minutes5 minutes1 minutes1
B2002,250 TFLOPS · $7.99/hr2.3$18.5017 minutes2 minutes1 minutes1

Compute = 6 × 125e6 parameters × tokens, at 40% utilisation of dense BF16 peak, with perfect scaling across GPUs. Memory for weights, gradients and Adam state is 2.0 GB before activations.

What real runs of this size reported

RunSize and dataReported
nanochatGPT-2 capability, 8 H100s, about 2 hoursabout $48

Published GPU-hours for Llama 3.1 came in 1.2–2.9× above the 6ND estimate; budget for that gap, plus experiments, failed runs and data work, before you treat these numbers as a quote.

Change the assumptions

Pre-filled for 125M parameters at 2.5B tokens on H100s.

Estimated cost

$6

H100 SXM-hours

1

Time on 8 GPUs

10 minutes

Compute

1.9e18 FLOPs

Full training needs about 2 GB for weights, gradients and optimiser state before activations, so at least 1 H100 SXM just to hold the model. Fine-tuning with QLoRA instead needs about 0 GB for 4-bit weights, which fits on a single 32 GB card.

Frequently asked questions

How much does it cost to train a 125M model?
About $5.78 on H100s at $4.39/hour for a compute-optimal run of 2.5B tokens (6 × parameters × tokens at 40% utilisation). At 10B tokens (80 tokens per parameter, typical of GPT-2-small reproductions) it is about $23.12. Treat both as floors: published runs used 1.2–2.9× more GPU-hours than the formula.
How long does it take to train a 125M model?
About 10 minutes on one 8-GPU H100 node for 2.5B tokens, assuming perfect scaling, or 1 minutes on 64 GPUs.
How much GPU memory does training a 125M model need?
About 2.0 GB for weights, gradients and Adam state (16 bytes per parameter) before activations, so at least 1 × 80 GB GPU with the state sharded.
Which GPU is cheapest for training a 125M model?
At on-demand November 2026 rates, the B200 ($7.99/hour) gives the lowest compute cost, about $4.62, by our estimate. Peak FLOPS aren't delivered performance, so benchmark your own code before a large run.

Fine-tuning instead

Open models of this size already carry far more pretraining than most budgets allow. Adapting one costs a tiny fraction:

The method in depth is in the free Academy course The Economics of Model Training and the article What it really costs to train an LLM; terms are defined in the glossary under 6ND, Chinchilla scaling and MFU.

Sources

  1. [1]Kaplan et al., 2020. Scaling Laws for Neural Language Models (arXiv:2001.08361)
  2. [2]Hoffmann et al., 2022. Training Compute-Optimal Large Language Models (arXiv:2203.15556)
  3. [3]Meta, 2024. The Llama 3 Herd of Models (arXiv:2407.21783)
  4. [4]Rajbhandari et al., 2019. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (arXiv:1910.02054)
  5. [5]Runpod. GPU pricing (updated 12 November 2026)
  6. [6]Karpathy. nanochat

You probably don't need to pretrain.

Kojen's prompt tuning and model search run on hosted models against your own test cases, and text LoRA trains on real GPUs (for accounts with GPU training enabled) when you need weights to change.

Start training free