AI2026-05-17

MoE (Mixture of Experts)

Also known as: MoE / Mixture of Experts / 混合エキスパート

A model architecture with multiple specialized sub-networks (experts) where only a sparse subset is activated per token, allowing parameter counts to scale without proportionally increasing compute.

Overview

MoE places multiple specialized feed-forward sub-networks (experts) inside the Transformer, with a router selecting only a few experts per token. GPT-4, Mixtral, and Rakuten AI 3.0 use MoE. Total parameter count is large but activated parameters per token are small, keeping per-token compute lower than a comparably capable dense model.

MoE vs Dense Model

MoE achieves better FLOPs efficiency than dense at equivalent capability, but requires all experts to reside in memory simultaneously, raising VRAM requirements. Mistral Small 4 (119B MoE) illustrates this tradeoff: strong capability with high total parameter count.

A comprehensive analysis of Rakuten AI 3.0's Mixture of Experts architecture with 700B parameters. Explore the 8-expert configuration, 40B active parameter efficiency, and technical background behind achieving 8.88 on Japanese MT-Bench.

Mistral Small 4 Complete Guide — Unified Reasoning, Multimodal & Code in 119B MoE [2026]

Mistral Small 4, released March 2026, unifies reasoning, multimodal vision, and agentic coding in a 119B MoE model under Apache 2.0. Supports 11 languages including Japanese. Full specs, setup guide, and model comparisons.

MiniMax M2.5 Complete Guide — Lightning Attention Achieves 80.2% SWE-bench [2026]

MiniMax M2.5 achieves 80.2% on SWE-bench Verified using proprietary Lightning Attention in a 230B MoE model. Full breakdown of architecture, benchmarks, license terms, and setup instructions.

Kimi K2.5 Complete Guide — 1 Trillion Parameter MIT-Licensed Open-Source LLM [2026]

Kimi K2.5, released by Moonshot AI on January 27, 2026, is a 1 trillion parameter (32B active) MoE model under the MIT License. It scores 76.8% on SWE-bench, 99.0% on HumanEval, and 87.6% on GPQA Diamond. This guide covers its architecture, hardware requirements, Ollama setup, and practical use cases.

Mistral Large 4 Requirements: VRAM & GPU for the 1T MoE (FP8 ~1TB)

Mistral Large 4 (1T MoE, 49B active) needs an estimated ~1TB at FP8, ~500-580GB at 4-bit. GPU, Mac and API cost compared. Updated Oct 2026; weights due soon.

Reflection Beam Requirements: VRAM & GPU for the 501B MoE (FP8 ~501GB, 8x H100+)

Reflection Beam (501B MoE, 23B active) needs an estimated ~501GB at FP8, ~250-290GB at 4-bit. GPU, Mac and CPU-offload fits as of October 2026; weights not out.

Feel free to contact us