QLoRA
QLoRA is LoRA on top of a base model whose frozen weights are stored in 4-bit, cutting fine-tuning memory roughly four-fold compared with LoRA on 16-bit weights.
Dettmers et al. introduced the 4-bit NormalFloat (NF4) data type, double quantisation of the scaling constants (about 4.127 bits per weight in total) and paged optimisers, and fine-tuned a 65B model on a single 48 GB GPU.
Weights are dequantised on the fly for each matrix multiplication, so QLoRA is somewhat slower than LoRA on the same GPU; it trades speed for memory.
Learn more on Kojen
- Academy: LoRA and QLoRA
- Reference: VRAM requirements by model
- Reference: fine-tuning and training costs
Related terms
- LoRALoRA (low-rank adaptation) fine-tunes a model by freezing its weights and training small low-rank matrices added to chosen layers, so only a tiny fraction of parameters is updated..
- QuantizationQuantization stores a model's weights (and sometimes activations) in fewer bits, such as 8 or 4 instead of 16, to cut memory and often speed up inference at a small cost in accuracy..
- KV cacheThe KV cache stores the attention keys and values of tokens a Transformer has already processed, so generating each new token doesn't recompute them.
Sources
- Dettmers et al., 2023. QLoRA: Efficient Finetuning of Quantized LLMs (arXiv:2305.14314)
Put the definitions to work.
Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.
Start training free