If you're adapting an open model, you'll choose between three ways of updating it. Full fine-tuning changes every weight. LoRA freezes the model and trains small add-on matrices. QLoRA does the same on a model compressed to 4 bits. The choice decides which GPU you need, what a run costs and, sometimes, how good the result is.
Here is the arithmetic for each method on an 8B model, Qwen3-8B, worked from its published architecture, followed by what the research says about quality.
What each method changes
Full fine-tuning updates all the parameters. Every task gets its own full copy of the model, and training needs memory for the weights, their gradients and the optimiser's state.
LoRA (low-rank adaptation) freezes the pretrained weights and adds, beside a chosen weight matrix W, two small matrices B and A whose product BA has rank r. Only A and B are trained. For GPT-3 175B, the LoRA paper reported 10,000 times fewer trainable parameters and three times less GPU memory than full fine-tuning, with no added inference latency because BA can be merged back into W.[1]
QLoRA stores the frozen base in a 4-bit format (NormalFloat4), quantises the quantisation constants themselves (“double quantisation”), pages optimiser state to CPU memory during spikes, and backpropagates through the frozen 4-bit weights into LoRA adapters. Its authors fine-tuned a 65B model on a single 48 GB GPU while matching 16-bit fine-tuning performance on their benchmarks.[2]
How many parameters LoRA trains
Qwen3-8B has 8.2B parameters across 36 layers, with grouped-query attention (32 query heads, 8 key-value heads).[3] Applying LoRA to all seven projection matrices in every layer, which is what recent work recommends,[4] gives:
Qwen3-8B: hidden 4,096, 32 query heads + 8 KV heads × 128 dims, MLP 12,288, 36 layers LoRA on a d_in × d_out matrix adds r × (d_in + d_out) parameters q_proj 4,096 → 4,096 r × 8,192 k_proj 4,096 → 1,024 r × 5,120 v_proj 4,096 → 1,024 r × 5,120 o_proj 4,096 → 4,096 r × 8,192 gate 4,096 → 12,288 r × 16,384 up 4,096 → 12,288 r × 16,384 down 12,288 → 4,096 r × 16,384 per layer r × 75,776 r = 16, 36 layers: 16 × 75,776 × 36 = 43.6M trainable parameters (0.53% of 8.2B) adapter file in bf16: 43.6M × 2 bytes ≈ 87 MB (the full model is ≈ 16.4 GB)
The same settings on Llama-3.1-8B-Instruct (32 layers, MLP width 14,336) give about 41.9M parameters. At rank 64 either model trains roughly four times as many, still under 2.5% of the base.
How much memory each method needs
Mixed-precision Adam keeps 16-bit weights and gradients plus 32-bit master weights and two 32-bit moments: 16 bytes per trained parameter. A frozen weight costs only its storage. That gives these estimates for Qwen3-8B, before activations:
Full fine-tuning (bf16 + Adam, fp32 master weights)
8.2B × 16 bytes ≈ 131 GB + activations
LoRA (frozen bf16 base)
8.2B × 2 bytes ≈ 16.4 GB
43.6M adapter params × 16 bytes ≈ 0.7 GB
≈ 17 GB + activations
QLoRA (frozen 4-bit NF4 base)
8.2B × 0.5 bytes ≈ 4.1 GB (rule of thumb)
embeddings + lm_head kept in 16-bit:
2 × 151,936 × 4,096 = 1.24B params × 2 B ≈ 2.5 GB
remaining 6.95B × 0.5 B ≈ 3.5 GB
+ adapters ≈ 0.7 GB
≈ 6.7 GB + activationsThe QLoRA line shows why rules of thumb undercount small models. Libraries usually leave the embedding table and output head unquantised, and with a 152K-token vocabulary those two matrices are 15% of Qwen3-8B's parameters. Activations come on top of every figure and grow with batch size and sequence length; gradient checkpointing trades extra compute for much less activation memory. Our fine-tuning VRAM calculator runs the rule-of-thumb version for any model size and GPU.
| Method | Memory before activations | Smallest practical setup |
|---|---|---|
| Full fine-tuning | ≈ 131 GB | Several 80 GB GPUs with sharded optimiser state |
| LoRA, bf16 base | ≈ 17 GB | One 24–48 GB GPU; an 80 GB card for long sequences |
| QLoRA, 4-bit base | ≈ 7 GB | One 16–24 GB GPU |
Speed and cost
Per token, all three methods do roughly the same forward and backward passes through the whole network, so compute is in the same range. LoRA skips the weight gradients for the frozen matrices, but gradient checkpointing adds back a second forward pass. QLoRA is usually slower per token than LoRA, because it dequantises weights on the fly.
The cost difference comes from hardware. Full fine-tuning of an 8B model needs a multi-GPU node; LoRA and QLoRA need one card. We work through the dollar figures in what it costs to fine-tune an 8B model: a few thousand examples on a rented A100 or H100 costs a few dollars of GPU time.
Does LoRA match full fine-tuning?
Sometimes, and the research now says fairly precisely when.
- When it falls short. Biderman et al. found LoRA substantially underperformed full fine-tuning on code and maths when training on large amounts of new data, and that full fine-tuning learns weight changes with a rank 10–100 times higher than typical LoRA settings. LoRA also forgot less of what the base model knew outside the target domain.[5]
- When it matches. Thinking Machines' “LoRA Without Regret” found LoRA matches full fine-tuning when it's applied to every layer, especially the MLP layers, and when the dataset doesn't exceed the adapter's capacity, which typical post-training datasets don't. Attention-only LoRA learned more slowly even at matched parameter count.[4]
- Hyperparameters differ. The same study found LoRA's best learning rate is about ten times full fine-tuning's, and that LoRA tolerates large batch sizes less well.[4]
- QLoRA's 4-bit base costs little. The QLoRA paper's 4-bit NF4 runs with double quantisation matched 16-bit LoRA and full fine-tuning on its benchmarks.[2]
Which to choose
| You want to… | Choose |
|---|---|
| Teach a format, style, tool-calling pattern or narrow task | LoRA |
| Do the same on the smallest GPU you can rent or own | QLoRA |
| Serve many task variants from one base model | LoRA (swap adapters per request) |
| Absorb billions of tokens of new domain data | Full fine-tuning, or LoRA at high rank |
| Teach new facts the model should cite | Neither: use retrieval |
That last row matters more than the others. Fine-tuning shapes behaviour well and inserts facts poorly; see fine-tuning vs RAG.
The LoRA and QLoRA lesson covers the method with a quiz. Note that r is not a quality dial: a higher rank adds capacity you only need for large datasets.



