Mixed precision
Mixed-precision training runs most arithmetic in 16-bit floating point (FP16 or BF16) while keeping a 32-bit master copy of the weights for updates, roughly halving memory traffic and using fast tensor cores.
Micikevicius et al. introduced the recipe with loss scaling for FP16; BF16's wider exponent range usually makes loss scaling unnecessary.
With Adam it costs about 16 bytes per parameter: 2 for 16-bit weights, 2 for gradients, 4 for the fp32 master weights and 8 for two optimiser moments, the figure ZeRO's analysis uses.
Learn more on Kojen
Related terms
- QuantizationQuantization stores a model's weights (and sometimes activations) in fewer bits, such as 8 or 4 instead of 16, to cut memory and often speed up inference at a small cost in accuracy..
- Gradient checkpointingGradient checkpointing saves memory in training by keeping only some layers' activations during the forward pass and recomputing the rest during the backward pass..
- MFUModel FLOPs utilisation (MFU) is the share of a GPU's theoretical peak throughput that a training run spends on the model's own arithmetic.
Sources
- Micikevicius et al., 2017. Mixed Precision Training (arXiv:1710.03740)
- Rajbhandari et al., 2019. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (arXiv:1910.02054)
Put the definitions to work.
Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.
Start training free