Edge0-35B-A3B Requirements: Run 35B MoE in Under 3GB

Edge0-35B-A3B-preview is an open-weight model that Edge0-AI released in early September 2026. It's built on Qwen3.5-MoE 35B-A3B (35B total parameters, roughly 3B active per token) at 4-bit quantization. Normally you'd need to keep the full ~19.6GB of weights resident in memory, but Edge0 instead keeps all the weights on SSD and streams only the experts each token needs, holding active memory under 3GiB. On a Mac mini M4 Pro (24GB unified memory), measured decode speed is roughly 14.9-17.7 tok/s, with prefill at roughly 113 tok/s cold and 140 tok/s warm. It ships only as an MLX (Apple Silicon) implementation; as of September 2026, CUDA support is still a roadmap item.

To compare memory needs against other models on your own GPU or Mac, our free VRAM calculator (no signup required) lets you pick a model, quantization, and context length. Note that the calculator uses the standard formula for loading full weights, so it does not capture Edge0's SSD-streaming trick (more on that below).

In a normal mixture-of-experts (MoE) model, which experts fire changes token by token, so every expert's weights have to sit resident in memory even though only a subset fires on any given token. As explained in our K2 Horizon requirements article, that "light on compute, heavy on memory" asymmetry holds even for a normal load of Qwen3.5-MoE 35B-A3B, the same base model Edge0 is built on. What Edge0 changes is that premise itself: it keeps all 256 experts' weights on SSD rather than in RAM, and a prediction head called the prerouter guesses ahead of time which 4 experts the next token will need, streaming just that slice into a RAM active buffer. The other 252 experts never load into RAM, so resident memory tracks the ~3B active count instead of the 35B total — under 3GiB.

If Edge0 read experts from SSD fresh for every token, that read latency would show up directly as inference lag. To avoid that, it uses a lightweight prediction head called the prerouter that predicts token t's routing decision one step early, at token t-1. As soon as that prediction is made, an async read starts pulling the weights from SSD in the background, so by the time compute for token t-1 finishes, the weights token t needs are already sitting in RAM. This design reportedly improves decode throughput by up to 59% compared to reading synchronously with no prerouting. When the prediction misses, the missing expert is fetched from SSD on the spot — slower, but generation continues.

Alongside SSD streaming, the other key piece is Recover-LoRA. The int4-quantized base model is frozen, and LoRA adapters are trained by distillation from an FP-precision teacher model, recovering most of the quality lost to 4-bit quantization. Edge0's own reporting states that, after int4 quantization, evaluation scores drop by an average of only about 3.9 points versus the fp16 base (reported figures: 86.6 on AIME 2026, 90.9 on HumanEval, 79.8 on GPQA-Diamond, and 81.0 on MMLU-Pro). These numbers come from figures Edge0 has published; we have not independently reproduced them.

- Storage: Official documentation says the architecture "assumes NVMe or comparable internal flash," but as of September 2026 no concrete required read speed (in GB/s) has been published. One community estimate puts the theoretical need at over 20GB/s if every token read cold off the drive, but in practice the OS file (page) cache serves most reads, so the actual sustained throughput needed appears to be lower. No test results for external SSDs or slower USB storage have been confirmed
- RAM (unified memory): Peak active memory is about 2.9GiB for short contexts. KV cache consumes RAM separately and grows with context length (the sibling Edge0-8B model reportedly measures ~3.3GiB total, including KV cache, at a 3.3k-token context). Treat 2.9GiB as a floor rather than a fixed figure when working with long prompts
- Supported hardware: As of September 2026, only the Apple Silicon MLX backend is supported. CUDA (NVIDIA GPUs) and general Windows/Linux support are roadmap items and do not work today
- Tested devices: Published information confirms measurements only on a Mac mini M4 Pro (24GB unified memory). The project claims support across M1-M4 generation Apple Silicon broadly, but per-device benchmarks are limited

Edge0 is a Python package published by Edge0-AI. It fetches the full model package (base checkpoint plus LoRA adapter plus prerouter weights) from Hugging Face in one step and launches a chat session via the CLI.

To try a development checkout of the repository directly, an editable install is also documented.

The Edge0 framework itself is designed as an extensible generalization of the "SSD expert offload + Recover-LoRA + prerouter routing prediction" recipe, and ships with two models out of the box: the 35B model covered here (Qwen3.5-MoE, K=4), plus the smaller Edge0-8B (Ling 3.0 hybrid, K=8, ~1.0GiB peak active memory). Backend code lives under edge0/backends/mlx/, structured to leave room for a future CUDA backend.

Edge0 isn't the first project to run an MoE model through SSD. llama.cpp can handle GGUF models via mmap, letting oversized models run with the OS handling paging even when they don't fit in physical memory. But OS-level page faults don't know what's needed next, so reads happen blindly, one blocking page at a time. Colibri, by contrast, is a C-based inference engine built to run the 370GB-class GLM 5.2 model on 25GB of RAM: it predicts router decisions ahead of time, requests the needed experts asynchronously, and speculatively preloads experts likely to be needed in the next layer (reported at roughly 72% prediction accuracy). The idea is close to Edge0's prerouter, but Colibri targets a general-purpose MoE model with 21,500+ fine-grained experts, and its measured throughput — roughly 0.1-1 tok/s — runs considerably slower than Edge0's.

What the comparison shows is that Edge0 is a purpose-built implementation: it trains a prerouter and a Recover-LoRA specifically for one model at a time (Qwen3.5-MoE 35B-A3B, with Edge0-8B added later), rather than aiming for the general applicability of llama.cpp's mmap offload or Colibri. That narrower scope trades away generality, but the per-model tuning of the prediction head and quality-recovery LoRA puts its measured throughput among the faster SSD-streaming approaches we could confirm publicly at the time of writing.

- It's a preview release: As the "preview" tag suggests, this isn't a final release. The README explicitly notes weak tool use, multi-step agentic planning, and long-horizon autonomy
- Apple Silicon only: As of September 2026, only the MLX backend runs. CUDA and Windows/Linux support are roadmap items and don't work yet
- KV cache grows with context: The 2.9GiB peak active memory figure is for short contexts. Long prompts or extended conversation history add to RAM usage via the KV cache
- No official storage-speed spec published: A concrete required read speed (in GB/s) isn't documented, and no test results exist for external or slower storage. Using anything other than internal NVMe means testing it yourself at your own risk

- What Is colibri? — The Pure-C Engine That Runs a 744B MoE Model on 25GB RAM
- MiniCPM5-2B Requirements: RAM, VRAM and GPU for Phones, Raspberry Pi and Laptops (2026)
- Local LLM Context Length and VRAM: KV Cache Formula Guide