MoE
A mixture-of-experts (MoE) model replaces each dense feed-forward layer with many expert networks and a router that sends each token to a few of them, so only a fraction of the parameters work on any token.
Shazeer et al. introduced the sparsely gated MoE layer; Switch Transformers routed each token to a single expert. Mixtral 8x7B has 46.7B parameters but uses 12.9B per token.
Compute and speed follow the active parameters, but memory follows the total, because any expert may be needed for the next token. That is why a 30B-A3B model trains cheaply per token yet needs a 30B model's memory.
Learn more on Kojen
Related terms
- 6ND6ND is the rule of thumb that training a Transformer takes about 6 floating-point operations per parameter per token, so compute ≈ 6 × N (parameters) × D (training tokens)..
- KV cacheThe KV cache stores the attention keys and values of tokens a Transformer has already processed, so generating each new token doesn't recompute them.
- LoRALoRA (low-rank adaptation) fine-tunes a model by freezing its weights and training small low-rank matrices added to chosen layers, so only a tiny fraction of parameters is updated..
Sources
- Shazeer et al., 2017. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (arXiv:1701.06538)
- Fedus et al., 2021. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (arXiv:2101.03961)
- Jiang et al., 2024. Mixtral of Experts (arXiv:2401.04088)
Put the definitions to work.
Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.
Start training free