How the estimate works
Memory is mostly bytes per parameter times parameter count. Full fine-tuning with Adam keeps 16-bit weights and gradients plus 32-bit master weights and two optimiser moments, about 16 bytes per parameter. LoRA freezes the base model, so only its 16-bit weights (2 bytes) stay resident, and QLoRA stores them in 4-bit (half a byte). Activations and the KV cache come on top, which is what the headroom setting covers.
To see what a training run costs once it fits, use the LLM training cost calculator. For buying versus renting a GPU, read What LLM can you train on a $10,000 computer?, and for the method in depth, the free Academy lesson on training memory and GPU tiers.
Frequently asked questions
- How much VRAM do you need to fine-tune a 7B model?
- About 112 GB for full fine-tuning with Adam (16 bytes per parameter), about 14 GB for LoRA on 16-bit weights, and about 3.5 GB for the 4-bit weights in QLoRA, before activations. A 7B QLoRA fine-tune fits comfortably on a single 24–32 GB card.
- Why does full fine-tuning need 16 bytes per parameter?
- Mixed-precision training with Adam keeps 16-bit weights and gradients (4 bytes) plus a 32-bit master copy of the weights and two 32-bit optimiser moments (12 bytes). Activations come on top and grow with batch size and sequence length.
- What is the difference between LoRA and QLoRA?
- Both freeze the base model and train small added matrices. LoRA keeps the frozen weights in 16-bit; QLoRA quantises them to 4-bit, cutting their memory by about four times. The QLoRA paper fine-tuned a 65B model on a single 48 GB GPU.
- Can you fine-tune a 70B model on one GPU?
- Not on a 32 GB consumer card: its 4-bit weights alone take about 35 GB. With QLoRA it fits on one 80 GB H100 or a 180 GB B200, with room for activations.
- Why leave headroom?
- The figures cover weights and optimiser state only. Activations, the KV cache, CUDA kernels and memory fragmentation all need space too, so plan for at least 20% more than the estimate, and more for long sequences or large batches.