Skip to content

Training

LoRA vs QLoRA vs full fine-tuning: memory, cost and quality

What each method trains, worked memory arithmetic for Qwen3-8B (131 GB, 17 GB, about 7 GB), how many parameters a rank-16 adapter adds, and when LoRA matches full fine-tuning.

By Will Schulz9 min read
Three dark metal blocks of decreasing height, each with a thin glowing red glass sheet slotted into its top

If you're adapting an open model, you'll choose between three ways of updating it. Full fine-tuning changes every weight. LoRA freezes the model and trains small add-on matrices. QLoRA does the same on a model compressed to 4 bits. The choice decides which GPU you need, what a run costs and, sometimes, how good the result is.

Here is the arithmetic for each method on an 8B model, Qwen3-8B, worked from its published architecture, followed by what the research says about quality.

What each method changes

Full fine-tuning updates all the parameters. Every task gets its own full copy of the model, and training needs memory for the weights, their gradients and the optimiser's state.

LoRA (low-rank adaptation) freezes the pretrained weights and adds, beside a chosen weight matrix W, two small matrices B and A whose product BA has rank r. Only A and B are trained. For GPT-3 175B, the LoRA paper reported 10,000 times fewer trainable parameters and three times less GPU memory than full fine-tuning, with no added inference latency because BA can be merged back into W.[1]

QLoRA stores the frozen base in a 4-bit format (NormalFloat4), quantises the quantisation constants themselves (“double quantisation”), pages optimiser state to CPU memory during spikes, and backpropagates through the frozen 4-bit weights into LoRA adapters. Its authors fine-tuned a 65B model on a single 48 GB GPU while matching 16-bit fine-tuning performance on their benchmarks.[2]

How many parameters LoRA trains

Qwen3-8B has 8.2B parameters across 36 layers, with grouped-query attention (32 query heads, 8 key-value heads).[3] Applying LoRA to all seven projection matrices in every layer, which is what recent work recommends,[4] gives:

Qwen3-8B: hidden 4,096, 32 query heads + 8 KV heads × 128 dims, MLP 12,288, 36 layers

LoRA on a d_in × d_out matrix adds r × (d_in + d_out) parameters
q_proj  4,096 → 4,096    r × 8,192
k_proj  4,096 → 1,024    r × 5,120
v_proj  4,096 → 1,024    r × 5,120
o_proj  4,096 → 4,096    r × 8,192
gate    4,096 → 12,288   r × 16,384
up      4,096 → 12,288   r × 16,384
down   12,288 → 4,096    r × 16,384
per layer                r × 75,776

r = 16, 36 layers: 16 × 75,776 × 36 = 43.6M trainable parameters (0.53% of 8.2B)
adapter file in bf16: 43.6M × 2 bytes ≈ 87 MB   (the full model is ≈ 16.4 GB)

The same settings on Llama-3.1-8B-Instruct (32 layers, MLP width 14,336) give about 41.9M parameters. At rank 64 either model trains roughly four times as many, still under 2.5% of the base.

How much memory each method needs

Mixed-precision Adam keeps 16-bit weights and gradients plus 32-bit master weights and two 32-bit moments: 16 bytes per trained parameter. A frozen weight costs only its storage. That gives these estimates for Qwen3-8B, before activations:

Full fine-tuning (bf16 + Adam, fp32 master weights)
  8.2B × 16 bytes                           ≈ 131 GB  + activations

LoRA (frozen bf16 base)
  8.2B × 2 bytes                            ≈ 16.4 GB
  43.6M adapter params × 16 bytes           ≈  0.7 GB
                                            ≈ 17 GB   + activations

QLoRA (frozen 4-bit NF4 base)
  8.2B × 0.5 bytes                          ≈  4.1 GB  (rule of thumb)
  embeddings + lm_head kept in 16-bit:
  2 × 151,936 × 4,096 = 1.24B params × 2 B  ≈  2.5 GB
  remaining 6.95B × 0.5 B                   ≈  3.5 GB
  + adapters                                ≈  0.7 GB
                                            ≈ 6.7 GB  + activations

The QLoRA line shows why rules of thumb undercount small models. Libraries usually leave the embedding table and output head unquantised, and with a 152K-token vocabulary those two matrices are 15% of Qwen3-8B's parameters. Activations come on top of every figure and grow with batch size and sequence length; gradient checkpointing trades extra compute for much less activation memory. Our fine-tuning VRAM calculator runs the rule-of-thumb version for any model size and GPU.

MethodMemory before activationsSmallest practical setup
Full fine-tuning≈ 131 GBSeveral 80 GB GPUs with sharded optimiser state
LoRA, bf16 base≈ 17 GBOne 24–48 GB GPU; an 80 GB card for long sequences
QLoRA, 4-bit base≈ 7 GBOne 16–24 GB GPU

Speed and cost

Per token, all three methods do roughly the same forward and backward passes through the whole network, so compute is in the same range. LoRA skips the weight gradients for the frozen matrices, but gradient checkpointing adds back a second forward pass. QLoRA is usually slower per token than LoRA, because it dequantises weights on the fly.

The cost difference comes from hardware. Full fine-tuning of an 8B model needs a multi-GPU node; LoRA and QLoRA need one card. We work through the dollar figures in what it costs to fine-tune an 8B model: a few thousand examples on a rented A100 or H100 costs a few dollars of GPU time.

Does LoRA match full fine-tuning?

Sometimes, and the research now says fairly precisely when.

  • When it falls short. Biderman et al. found LoRA substantially underperformed full fine-tuning on code and maths when training on large amounts of new data, and that full fine-tuning learns weight changes with a rank 10–100 times higher than typical LoRA settings. LoRA also forgot less of what the base model knew outside the target domain.[5]
  • When it matches. Thinking Machines' “LoRA Without Regret” found LoRA matches full fine-tuning when it's applied to every layer, especially the MLP layers, and when the dataset doesn't exceed the adapter's capacity, which typical post-training datasets don't. Attention-only LoRA learned more slowly even at matched parameter count.[4]
  • Hyperparameters differ. The same study found LoRA's best learning rate is about ten times full fine-tuning's, and that LoRA tolerates large batch sizes less well.[4]
  • QLoRA's 4-bit base costs little. The QLoRA paper's 4-bit NF4 runs with double quantisation matched 16-bit LoRA and full fine-tuning on its benchmarks.[2]

Which to choose

You want to…Choose
Teach a format, style, tool-calling pattern or narrow taskLoRA
Do the same on the smallest GPU you can rent or ownQLoRA
Serve many task variants from one base modelLoRA (swap adapters per request)
Absorb billions of tokens of new domain dataFull fine-tuning, or LoRA at high rank
Teach new facts the model should citeNeither: use retrieval

That last row matters more than the others. Fine-tuning shapes behaviour well and inserts facts poorly; see fine-tuning vs RAG.

The LoRA and QLoRA lesson covers the method with a quiz. Note that r is not a quality dial: a higher rank adds capacity you only need for large datasets.

Frequently asked questions

What is the difference between LoRA and QLoRA?
Both freeze the base model and train small low-rank adapter matrices. LoRA keeps the frozen weights in 16-bit; QLoRA stores them in 4-bit NormalFloat, cutting their memory about fourfold, and backpropagates through them into the adapters.
How much GPU memory does it take to fine-tune an 8B model?
For Qwen3-8B before activations: about 131 GB for full fine-tuning with Adam, about 17 GB for LoRA on a bf16 base, and about 7 GB for QLoRA once the 16-bit embeddings and output head are counted.
Is LoRA as good as full fine-tuning?
For typical post-training datasets, it can be: Thinking Machines found LoRA matches full fine-tuning when applied to all layers including the MLPs, with about a 10× higher learning rate. On very large new-domain datasets such as code and maths, Biderman et al. found LoRA learns less but forgets less.
How many parameters does a LoRA adapter add?
Rank r on a d_in × d_out matrix adds r × (d_in + d_out). For Qwen3-8B at rank 16 on all seven projections in 36 layers, that's 43.6 million parameters, 0.53% of the model, or an 87 MB adapter file in bf16.
Does QLoRA reduce quality?
Little, in the original paper: 4-bit NF4 with double quantisation matched 16-bit LoRA and full fine-tuning on its benchmarks, and fine-tuned a 65B model on one 48 GB GPU. It is usually slower per token because weights are dequantised on the fly.

Go deeper in the Academy

Free lessons with quizzes, citations and a hands-on exercise.

Sources

  1. [1]LoRA: Low-Rank Adaptation of Large Language Models. Hu et al., arXiv (2021).
  2. [2]QLoRA: Efficient Finetuning of Quantized LLMs. Dettmers et al., arXiv (2023).
  3. [3]Qwen3-8B model card and config. Qwen, Hugging Face.
  4. [4]LoRA Without Regret. Schulman et al., Thinking Machines Lab (29 September 2025).
  5. [5]LoRA Learns Less and Forgets Less. Biderman et al., arXiv (2024).

Put it into practice.

Kojen gives you the Forge, LLM-judged evaluations, red-team simulations and live endpoints in one place. Free to start, no card needed, then pay as you go.

Start training free

More reading

All articles →