Skip to content

Glossary · Training

LoRA

LoRA (low-rank adaptation) fine-tunes a model by freezing its weights and training small low-rank matrices added to chosen layers, so only a tiny fraction of parameters is updated.

Also called Low-Rank Adaptation · Reviewed 15 November 2026

For a frozen weight matrix W, LoRA learns B·A with rank r much smaller than W's dimensions, and uses W + B·A in the forward pass. Hu et al. cut GPT-3 175B's trainable parameters 10,000-fold and its GPU memory 3-fold, with quality on par with full fine-tuning on their tasks.

Because gradients and optimiser state exist only for the adapters, memory is dominated by the frozen 16-bit weights, about 2 bytes per parameter. Adapters are small files that can be swapped on one base model.

Formula

h = W·x + (α / r)·B·A·x,   A ∈ ℝ^(r×d), B ∈ ℝ^(d×r)

Learn more on Kojen

Sources

  1. [1]Hu et al., 2021. LoRA: Low-Rank Adaptation of Large Language Models (arXiv:2106.09685)

All 31 terms

Put the definitions to work.

Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.

Start training free