Skip to content

Glossary · Efficiency

Quantization

Quantization stores a model's weights (and sometimes activations) in fewer bits, such as 8 or 4 instead of 16, to cut memory and often speed up inference at a small cost in accuracy.

Also called quantisation, int8, int4, 4-bit · Reviewed 15 November 2026

LLM.int8() showed 8-bit inference without degradation by handling outlier features in 16-bit; GPTQ quantises weights to 3–4 bits post-training with little perplexity loss. 4-bit formats with group-wise scales take about 4.5 bits per weight in practice.

QLoRA trains adapters on 4-bit weights. Quality loss grows at lower bit widths and varies by model and task, so evaluate the quantised model, not just the original.

Learn more on Kojen

Sources

  1. [1]Dettmers et al., 2022. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (arXiv:2208.07339)
  2. [2]Frantar et al., 2022. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (arXiv:2210.17323)
  3. [3]Dettmers et al., 2023. QLoRA: Efficient Finetuning of Quantized LLMs (arXiv:2305.14314)

All 31 terms

Put the definitions to work.

Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.

Start training free