Skip to content

Glossary · Efficiency

MoE

A mixture-of-experts (MoE) model replaces each dense feed-forward layer with many expert networks and a router that sends each token to a few of them, so only a fraction of the parameters work on any token.

Also called Mixture of experts, sparse MoE · Reviewed 15 November 2026

Shazeer et al. introduced the sparsely gated MoE layer; Switch Transformers routed each token to a single expert. Mixtral 8x7B has 46.7B parameters but uses 12.9B per token.

Compute and speed follow the active parameters, but memory follows the total, because any expert may be needed for the next token. That is why a 30B-A3B model trains cheaply per token yet needs a 30B model's memory.

Learn more on Kojen

Sources

  1. [1]Shazeer et al., 2017. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (arXiv:1701.06538)
  2. [2]Fedus et al., 2021. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (arXiv:2101.03961)
  3. [3]Jiang et al., 2024. Mixtral of Experts (arXiv:2401.04088)

All 31 terms

Put the definitions to work.

Kojen evaluates, tunes and red-teams your agents with these methods built in: Wilson intervals, pass^k, LLM judges with visible components. Free to start.

Start training free