KV cache
The KV cache stores the attention keys and values of tokens a Transformer has already processed, so generating each new token doesn't recompute them. Its size grows linearly with context length and batch size.
Per token it takes 2 × layers × KV heads × head dimension × bytes per value: about 128 KiB in fp16 for Llama 3.1 8B, so a 128K-token context needs about 17 GB on top of the weights. Grouped-query attention shrinks it by sharing KV heads across query heads.
For long contexts or many concurrent users, the KV cache, not the weights, usually decides how much GPU memory inference needs.
Formula
bytes per token = 2 · layers · kv_heads · head_dim · bytes_per_value
Learn more on Kojen
Related terms
- QuantizationQuantization stores a model's weights (and sometimes activations) in fewer bits, such as 8 or 4 instead of 16, to cut memory and often speed up inference at a small cost in accuracy..
- Context engineeringContext engineering is designing everything that goes into a model's context window, instructions, tools, retrieved documents, memory and history, so an agent has the information it needs and little else..
- MoEA mixture-of-experts (MoE) model replaces each dense feed-forward layer with many expert networks and a router that sends each token to a few of them, so only a fraction of the parameters work on any token..
Sources
- Pope et al., 2022. Efficiently Scaling Transformer Inference (arXiv:2211.05102)
- Ainslie et al., 2023. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (arXiv:2305.13245)
Put the definitions to work.
Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.
Start training free