LoRA
LoRA (low-rank adaptation) fine-tunes a model by freezing its weights and training small low-rank matrices added to chosen layers, so only a tiny fraction of parameters is updated.
For a frozen weight matrix W, LoRA learns B·A with rank r much smaller than W's dimensions, and uses W + B·A in the forward pass. Hu et al. cut GPT-3 175B's trainable parameters 10,000-fold and its GPU memory 3-fold, with quality on par with full fine-tuning on their tasks.
Because gradients and optimiser state exist only for the adapters, memory is dominated by the frozen 16-bit weights, about 2 bytes per parameter. Adapters are small files that can be swapped on one base model.
Formula
h = W·x + (α / r)·B·A·x, A ∈ ℝ^(r×d), B ∈ ℝ^(d×r)
Learn more on Kojen
- Academy: LoRA and QLoRA
- Blog: Prompt tuning or fine-tuning?
- Reference: VRAM requirements by model
- Tool: fine-tuning VRAM calculator
Related terms
- QLoRAQLoRA is LoRA on top of a base model whose frozen weights are stored in 4-bit, cutting fine-tuning memory roughly four-fold compared with LoRA on 16-bit weights..
- QuantizationQuantization stores a model's weights (and sometimes activations) in fewer bits, such as 8 or 4 instead of 16, to cut memory and often speed up inference at a small cost in accuracy..
- Mixed precisionMixed-precision training runs most arithmetic in 16-bit floating point (FP16 or BF16) while keeping a 32-bit master copy of the weights for updates, roughly halving memory traffic and using fast tensor cores..
Sources
Put the definitions to work.
Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.
Start training free