Quantization
Quantization stores a model's weights (and sometimes activations) in fewer bits, such as 8 or 4 instead of 16, to cut memory and often speed up inference at a small cost in accuracy.
LLM.int8() showed 8-bit inference without degradation by handling outlier features in 16-bit; GPTQ quantises weights to 3–4 bits post-training with little perplexity loss. 4-bit formats with group-wise scales take about 4.5 bits per weight in practice.
QLoRA trains adapters on 4-bit weights. Quality loss grows at lower bit widths and varies by model and task, so evaluate the quantised model, not just the original.
Learn more on Kojen
Related terms
- QLoRAQLoRA is LoRA on top of a base model whose frozen weights are stored in 4-bit, cutting fine-tuning memory roughly four-fold compared with LoRA on 16-bit weights..
- KV cacheThe KV cache stores the attention keys and values of tokens a Transformer has already processed, so generating each new token doesn't recompute them.
- Mixed precisionMixed-precision training runs most arithmetic in 16-bit floating point (FP16 or BF16) while keeping a 32-bit master copy of the weights for updates, roughly halving memory traffic and using fast tensor cores..
Sources
- Dettmers et al., 2022. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (arXiv:2208.07339)
- Frantar et al., 2022. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (arXiv:2210.17323)
- Dettmers et al., 2023. QLoRA: Efficient Finetuning of Quantized LLMs (arXiv:2305.14314)
Put the definitions to work.
Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.
Start training free