Skip to content

Glossary · Efficiency

Gradient checkpointing

Gradient checkpointing saves memory in training by keeping only some layers' activations during the forward pass and recomputing the rest during the backward pass.

Also called activation checkpointing, activation recomputation · Reviewed 15 November 2026

Chen et al. showed it reduces activation memory from linear in depth to roughly its square root for about one extra forward pass of compute. Checkpointing every Transformer layer keeps just each layer's input.

Recomputing costs roughly one more forward pass, about a third more compute per step; selective recomputation (Korthikanti et al.) recomputes only the cheap but memory-hungry parts.

Learn more on Kojen

Sources

  1. [1]Chen et al., 2016. Training Deep Nets with Sublinear Memory Cost (arXiv:1604.06174)
  2. [2]Korthikanti et al., 2022. Reducing Activation Recomputation in Large Transformer Models (arXiv:2205.05198)

All 31 terms

Put the definitions to work.

Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.

Start training free