Skip to content

Glossary · Efficiency

Mixed precision

Mixed-precision training runs most arithmetic in 16-bit floating point (FP16 or BF16) while keeping a 32-bit master copy of the weights for updates, roughly halving memory traffic and using fast tensor cores.

Also called mixed-precision training, bf16 training · Reviewed 15 November 2026

Micikevicius et al. introduced the recipe with loss scaling for FP16; BF16's wider exponent range usually makes loss scaling unnecessary.

With Adam it costs about 16 bytes per parameter: 2 for 16-bit weights, 2 for gradients, 4 for the fp32 master weights and 8 for two optimiser moments, the figure ZeRO's analysis uses.

Learn more on Kojen

Sources

  1. [1]Micikevicius et al., 2017. Mixed Precision Training (arXiv:1710.03740)
  2. [2]Rajbhandari et al., 2019. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (arXiv:1910.02054)

All 31 terms

Put the definitions to work.

Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.

Start training free