Skip to content

Glossary · Efficiency

KV cache

The KV cache stores the attention keys and values of tokens a Transformer has already processed, so generating each new token doesn't recompute them. Its size grows linearly with context length and batch size.

Also called key-value cache · Reviewed 15 November 2026

Per token it takes 2 × layers × KV heads × head dimension × bytes per value: about 128 KiB in fp16 for Llama 3.1 8B, so a 128K-token context needs about 17 GB on top of the weights. Grouped-query attention shrinks it by sharing KV heads across query heads.

For long contexts or many concurrent users, the KV cache, not the weights, usually decides how much GPU memory inference needs.

Formula

bytes per token = 2 · layers · kv_heads · head_dim · bytes_per_value

Learn more on Kojen

Sources

  1. [1]Pope et al., 2022. Efficiently Scaling Transformer Inference (arXiv:2211.05102)
  2. [2]Ainslie et al., 2023. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (arXiv:2305.13245)

All 31 terms

Put the definitions to work.

Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.

Start training free