MoE (Mixture of Experts)

Overview

MoE places multiple specialized feed-forward sub-networks (experts) inside the Transformer, with a router selecting only a few experts per token. GPT-4, Mixtral, and Rakuten AI 3.0 use MoE. Total parameter count is large but activated parameters per token are small, keeping per-token compute lower than a comparably capable dense model.

MoE vs Dense Model

MoE achieves better FLOPs efficiency than dense at equivalent capability, but requires all experts to reside in memory simultaneously, raising VRAM requirements. Mistral Small 4 (119B MoE) illustrates this tradeoff: strong capability with high total parameter count.