Gradient checkpointing
Gradient checkpointing saves memory in training by keeping only some layers' activations during the forward pass and recomputing the rest during the backward pass.
Chen et al. showed it reduces activation memory from linear in depth to roughly its square root for about one extra forward pass of compute. Checkpointing every Transformer layer keeps just each layer's input.
Recomputing costs roughly one more forward pass, about a third more compute per step; selective recomputation (Korthikanti et al.) recomputes only the cheap but memory-hungry parts.
Learn more on Kojen
Related terms
- Mixed precisionMixed-precision training runs most arithmetic in 16-bit floating point (FP16 or BF16) while keeping a 32-bit master copy of the weights for updates, roughly halving memory traffic and using fast tensor cores..
- LoRALoRA (low-rank adaptation) fine-tunes a model by freezing its weights and training small low-rank matrices added to chosen layers, so only a tiny fraction of parameters is updated..
- KV cacheThe KV cache stores the attention keys and values of tokens a Transformer has already processed, so generating each new token doesn't recompute them.
Sources
- Chen et al., 2016. Training Deep Nets with Sublinear Memory Cost (arXiv:1604.06174)
- Korthikanti et al., 2022. Reducing Activation Recomputation in Large Transformer Models (arXiv:2205.05198)
Put the definitions to work.
Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.
Start training free