Memory first: what hardware can train what
Speed decides how long a run takes. Memory decides whether it runs at all, and training needs far more of it than running the same model.
In this lesson you'll
- Calculate the memory that full training needs
- Explain how LoRA and QLoRA lower that requirement
- Compare consumer and data-centre GPUs on memory, speed and price
Running a model mostly means holding its weights. Training also means holding gradients and optimiser state. With mixed-precision Adam, the standard setup, that comes to 16 bytes per parameter: 2 for the weights, 2 for the gradients and 12 for the optimiser [1]. Activations come on top, and they grow with batch size and sequence length.
| Model | Full training (16 bytes per parameter) | Weights in 4-bit (QLoRA) |
|---|---|---|
| 1B | 16 GB | 0.5 GB |
| 7B | 112 GB | 3.5 GB |
| 13B | 208 GB | 6.5 GB |
| 32B | 512 GB | 16 GB |
| 70B | 1,120 GB | 35 GB |
These figures leave out activations, so they're minimums. Fully training a 7B model already takes more memory than one 80 GB H100 has, which is why it needs several GPUs that split the training state between them, the approach ZeRO introduced [1].
Why fine-tuning fits on so much less
LoRA freezes the base model and trains small added matrices, so gradients and optimiser state only apply to a tiny share of the parameters [2]. QLoRA also stores the frozen weights in 4 bits, which let its authors fine-tune a 65B model on a single 48 GB GPU [3]. That's the difference between a multi-GPU cluster and one card.
The hardware tiers
| GPU | Memory | Dense BF16 peak | Rental, per hour |
|---|---|---|---|
| RTX 5090 | 32 GB [4] | 209.5 TFLOPS | $1.19 |
| H100 SXM | 80 GB [5] | about 989 TFLOPS | $3.99 |
| B200 | 180 GB [6] | 2,250 TFLOPS | $7.99 |
Rental prices are Runpod's Secure Cloud rates from 4 November 2026 [7]. The 96 GB RTX PRO 6000 sits between the tiers at $2.49 an hour, and it's a useful option when a job needs memory more than speed.
Per unit of peak compute, faster cards can cost less. An H100 has 4.7× the peak of an RTX 5090 at 3.4× the hourly price, and a B200 has about 2.3× the peak of an H100 at twice the price. Peak isn't delivered performance, so benchmark your own workload before you commit to a large run.
Interconnect matters at scale
Once a run spans many GPUs, they spend time exchanging data, and failures become routine. Over a 54-day snapshot of Llama 3 training, Meta logged 419 unexpected interruptions, about eight a day, and roughly 78% were confirmed or suspected hardware problems [8]. Checkpointing and automated recovery are part of the cost of large runs.
Key takeaways
- Full training needs about 16 bytes per parameter before activations.
- LoRA and QLoRA cut memory so much that a 65B model can be fine-tuned on one 48 GB GPU.
- Check memory first, then compare price per unit of real throughput.
Open the LoRA/QLoRA preview and look at the settings a real run would need. It's a preview with simulated metrics, so use it to learn the options, not to judge results.
Open the ForgeCheck your understanding
Score 70 or more to complete the lesson. Progress is saved in this browser.
Further reading
Sources
Checked against the original sources in November 2026.
- Rajbhandari, S. et al. (2019). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. SC 2020 · arXiv:1910.02054.
- Hu, E. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022 · arXiv:2106.09685.
- Dettmers, T. et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023 · arXiv:2305.14314.
- NVIDIA (2025). NVIDIA RTX Blackwell GPU Architecture. NVIDIA whitepaper.
- NVIDIA (2026). NVIDIA H100 Tensor Core GPU. nvidia.com.
- Lenovo Press (2026). NVIDIA HGX B200 product guide. lenovopress.lenovo.com.
- Runpod (2026). GPU pricing (updated 4 November 2026). runpod.io.
- Llama Team, AI @ Meta (2024). The Llama 3 Herd of Models. arXiv:2407.21783.