How much GPU memory each open model needs
Memory for fine-tuning and running 25 popular open-weights models, worked out from each model's published architecture. The figures use 2,048-token training sequences, LoRA rank 16 and an 8K-token inference context; each model's page has the full breakdown, a GPU-by-GPU verdict and a calculator for other settings.
| Model | Parameters | QLoRA | LoRA | Full fine-tune | Run, 16-bit | Run, 4-bit |
|---|---|---|---|---|---|---|
| Llama 3.1 8B | 8.03B | 9.4 GB | 20 GB | 132 GB | 17 GB | 5.6 GB |
| Llama 3.1 70B | 70.6B | 48 GB | 150 GB | 1,134 GB | 144 GB | 42 GB |
| Llama 3.2 1B | 1.24B | 3.7 GB | 5.1 GB | 22 GB | 2.7 GB | 1.0 GB |
| Llama 3.2 3B | 3.21B | 5.3 GB | 9.5 GB | 54 GB | 7.4 GB | 2.7 GB |
| Llama 3.3 70B | 70.6B | 48 GB | 150 GB | 1,134 GB | 144 GB | 42 GB |
| Qwen3 0.6B | 0.75B | 3.5 GB | 4.4 GB | 15 GB | 2.4 GB | 1.4 GB |
| Qwen3 1.7B | 2.03B | 4.7 GB | 7.2 GB | 35 GB | 5.0 GB | 2.1 GB |
| Qwen3 4B | 4.02B | 6.3 GB | 12 GB | 67 GB | 9.3 GB | 3.5 GB |
| Qwen3 8B | 8.19B | 10 GB | 21 GB | 135 GB | 18 GB | 5.8 GB |
| Qwen3 14B | 14.8B | 15 GB | 34 GB | 240 GB | 31 GB | 9.6 GB |
| Qwen3 32B | 32.8B | 26 GB | 72 GB | 529 GB | 68 GB | 21 GB |
| Qwen3 30B-A3B | 30.5B (3.3B active) | 20 GB | 64 GB | 492 GB | 62 GB | 18 GB |
| Qwen2.5 7B | 7.62B | 9.5 GB | 19 GB | 125 GB | 16 GB | 4.8 GB |
| Qwen2.5 72B | 72.7B | 51 GB | 155 GB | 1,169 GB | 148 GB | 44 GB |
| Mistral 7B | 7.25B | 6.3 GB | 17 GB | 117 GB | 16 GB | 5.2 GB |
| Mixtral 8x7B | 46.7B (12.9B active) | 26 GB | 95 GB | 749 GB | 94 GB | 27 GB |
| Gemma 3 4B | 4.30B | 8.6 GB | 14 GB | 74 GB | 8.9 GB | 2.7 GB |
| Gemma 3 12B | 12.2B | 14 GB | 31 GB | 200 GB | 25 GB | 7.7 GB |
| Gemma 3 27B | 27.4B | 24 GB | 63 GB | 445 GB | 56 GB | 17 GB |
| Phi-4 | 14.7B | 13 GB | 33 GB | 238 GB | 31 GB | 9.9 GB |
| DeepSeek-R1-Distill-Qwen-7B | 7.62B | 9.5 GB | 19 GB | 125 GB | 16 GB | 4.8 GB |
| DeepSeek-R1-Distill-Qwen-32B | 32.8B | 26 GB | 72 GB | 529 GB | 68 GB | 21 GB |
| DeepSeek-R1-Distill-Llama-70B | 70.6B | 48 GB | 150 GB | 1,134 GB | 144 GB | 42 GB |
| gpt-oss-20b | 20.9B (3.6B active) | 17 GB | 46 GB | 339 GB | 42 GB | 14 GB |
| gpt-oss-120b | 116.8B (5.1B active) | 66 GB | 238 GB | 1,873 GB | 234 GB | 66 GB |
By GPU
RTX 3090 (24 GB)
Llama 3.1 8B · Llama 3.2 1B · Llama 3.2 3B · Qwen3 4B · Qwen3 8B · Qwen3 14B · Mistral 7B · Gemma 3 12B · Phi-4 · gpt-oss-20b
RTX 4090 (24 GB)
Llama 3.1 8B · Llama 3.1 70B · Llama 3.2 1B · Llama 3.2 3B · Llama 3.3 70B · Qwen3 4B · Qwen3 8B · Qwen3 14B · Qwen3 32B · Qwen3 30B-A3B · Qwen2.5 7B · Mistral 7B · Mixtral 8x7B · Gemma 3 4B · Gemma 3 12B · Gemma 3 27B · Phi-4 · DeepSeek-R1-Distill-Qwen-7B · DeepSeek-R1-Distill-Qwen-32B · gpt-oss-20b
RTX 5090 (32 GB)
Llama 3.1 8B · Llama 3.1 70B · Qwen3 8B · Qwen3 14B · Qwen3 32B · Qwen3 30B-A3B · Qwen2.5 7B · Mistral 7B · Mixtral 8x7B · Gemma 3 12B · Gemma 3 27B · Phi-4 · DeepSeek-R1-Distill-Qwen-7B · DeepSeek-R1-Distill-Qwen-32B · gpt-oss-20b · gpt-oss-120b
L40S (48 GB)
Llama 3.1 8B · Qwen3 8B · Qwen3 14B · Qwen3 32B · Qwen3 30B-A3B · Gemma 3 27B
A100 40GB (40 GB)
A100 80GB (80 GB)
Llama 3.1 8B · Llama 3.1 70B · Qwen3 32B · Qwen2.5 72B · Mixtral 8x7B
H100 SXM (80 GB)
Llama 3.1 8B · Llama 3.1 70B · Llama 3.3 70B · Qwen3 8B · Qwen3 14B · Qwen3 32B · Qwen3 30B-A3B · Qwen2.5 72B · Mixtral 8x7B · Gemma 3 27B · DeepSeek-R1-Distill-Qwen-32B · DeepSeek-R1-Distill-Llama-70B · gpt-oss-20b · gpt-oss-120b
H200 (141 GB)
Llama 3.1 70B · Llama 3.3 70B · Qwen2.5 72B · DeepSeek-R1-Distill-Llama-70B · gpt-oss-120b
B200 (180 GB)
How we estimate it
Every figure comes from the model's published architecture (layers, hidden size, attention and KV heads, vocabulary) rather than a rule of thumb, using the same functions as the calculator above. Sizes are in GB (10⁹ bytes).
- Full fine-tuning: 16 bytes per parameter for mixed-precision Adam (bf16 weights and gradients, fp32 master weights and two moments), as in the ZeRO paper.
- LoRA: frozen bf16 weights (2 bytes per parameter) plus 16 bytes per adapter parameter, rank 16 on every linear layer. For mixture-of-experts models, adapters go on attention only and the experts stay frozen.
- QLoRA: NF4 weights with double quantisation (4.127 bits per parameter); embeddings and the output head stay in bf16, as bitsandbytes leaves them.
- Activations: gradient checkpointing and FlashAttention. Each layer's bf16 input is kept, one layer is recomputed at a time, and the output logits take 8 bytes per vocabulary entry per token (fp32 values and gradients). Chunked cross-entropy, as in Unsloth or Liger kernels, cuts that last term.
- Inference: 2, 1 or about 0.56 bytes per weight (4-bit with group-wise scales), plus an fp16 KV cache of 2 × layers × KV heads × head dim × 2 bytes per token; sliding-window layers only cache their window.
- Fits means the estimate is within 90% of the card's listed memory; the rest covers the CUDA context, kernels and fragmentation. Multi-GPU counts assume memory splits evenly.
More tools
- Fine-tuning VRAM calculatorAny parameter count
- Fine-tuning and training costsGPU-hours and dollars by model
- LLM training cost calculator6 × parameters × tokens
- GlossaryLoRA, QLoRA, KV cache, MoE and more
Sources
- Dettmers et al., 2023. QLoRA: Efficient Finetuning of Quantized LLMs (arXiv:2305.14314)
- Hu et al., 2021. LoRA: Low-Rank Adaptation of Large Language Models (arXiv:2106.09685)
- Rajbhandari et al., 2019. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (arXiv:1910.02054)
- Korthikanti et al., 2022. Reducing Activation Recomputation in Large Transformer Models (arXiv:2205.05198)
- Chen et al., 2016. Training Deep Nets with Sublinear Memory Cost (arXiv:1604.06174)
Most agents don't need a bigger GPU.
Kojen's prompt tuning and model search run for real on hosted models against your own test cases. Free to start, then pay as you go.
Start training free