DwarfStar ds4: antirez's Local LLM Engine for Mac & CUDA

DwarfStar (ds4, GitHub: antirez/ds4) is an MIT-licensed local inference engine from Salvatore Sanfilippo (antirez), the creator of Redis. It is deliberately narrow: instead of running any model, it targets a handful of open-weight models, namely DeepSeek V4/V4.1 Flash, GLM 5.2/5.3, and Qwen3.8 Flash Next, on 96-128 GB Macs (Metal), NVIDIA DGX Spark, Strix Halo, and multi-GPU CUDA boxes. It is not a general GGUF runner; it only runs the GGUF files the project itself produces. The repository was created on 2026-05-06 and has reached about 23,000 stars, and on 2026-10-03 the project site dwarfstar.sh hit the Hacker News front page under the title "From the creator of Redis; run LLM locally with ds4".

This article draws on the official docs as of October 2026 (README, MODELS, PERFORMANCE, CLIENTS, SSD_STREAMING) and covers what ds4 can do, supported hardware and memory needs, setup, connecting Claude Code, Codex CLI and OpenCode, measured speeds, how it differs from other tools, and caveats. Up front: the project itself calls the software beta quality.

The stated goal is to be the best way to run a few excellent large language models on consumer hardware, meaning hardware people can actually own. To get there, the project writes its own small native inference engine tuned for specific models. The first priority is DeepSeek V4 Flash (including the experimental vision model), joined by DeepSeek V4.1 Flash (Metal, plus text inference on CUDA), GLM 5.2 and 5.3, GLM 5.3 Flash, DeepSeek V4 PRO, and Qwen3.8 Flash Next (Metal and CUDA).

The code is self-contained, and model loading, prompt rendering, tool calls, KV state, the HTTP server, and the coding agent are built and tested together. It does not link against GGML, but it exists thanks to the kernels, quantization formats, and GGUF ecosystem developed by llama.cpp, and the README thanks Georgi Gerganov and the other contributors explicitly. Model support is also intentionally opportunistic: a model may be removed when a better replacement arrives. The sweet spot is 128 GB laptops and 256/512 GB workstations.

- Run very capable models on owned hardware: MacBooks, DGX Spark, Strix Halo and similar machines
- Asymmetric 2-bit quantization: the routed experts, which dominate the model, are compressed hard (IQ2_XXS gate/up, Q2_K down) while other components stay at higher precision such as Q8 and F16/F32
- SSD streaming: run models larger than RAM by caching hot experts and reading the rest from SSD
- Disk KV cache: save compatible KV snapshots so long prompts need not be recomputed
- Three ways to run: the interactive CLI ds4, the OpenAI/Anthropic-compatible ds4-server, and the native ds4-agent that talks to the engine directly
- Vision: PNG/JPEG input with a matching encoder passed via --vision
- MTP speculative decoding: --mtp uses the built-in draft block of GLM and Qwen
- Multi-Mac setups: tensor parallelism over RDMA with two 128 GB Macs, plus pipeline parallelism to pool RAM
- Multi-GPU CUDA: Ada Lovelace cards including the L40S are supported; an eight-L40S setup reached about 126 t/s aggregate generation with 16 sessions

The README's motivations: capable open-weight models now fit on high-end personal machines, DeepSeek V4 Flash/PRO and GLM 5.2 tolerate aggressive routed-expert quantization, and compressed KV caches plus fast local SSDs make long contexts practical.

There are three backends. Metal is the primary target, on Macs with 96 GB or more; smaller machines use SSD streaming. On NVIDIA CUDA, the DGX Spark is the main goal, and multi-GPU systems unsupported by other backends also work (for example DeepSeek V4 Flash on Ada Lovelace cards). ROCm targets Strix Halo systems such as the Framework Desktop. Build with make (Metal), make cuda-spark (DGX Spark), make strix-halo, or make cuda-generic.

Main download targets and sizes, per the official MODELS.md (runtime memory for context and buffers is on top):

To check quickly whether a given GPU or Mac can run a model, the VRAM Calculator (free, no sign-up) estimates the VRAM you need once you pick a model, quantization, and context length.

For the underlying models, see our guides on DeepSeek V4 requirements, VRAM and pricing, GLM 5.3 Flash requirements, and Qwen3.8 Flash Next requirements. ds4 is the runtime side of that story: how to actually run them on which machine.

Getting from source to a first run takes a few commands. Clone the repo and build for your platform (the example below is Apple Silicon). For a first run on a 96 or 128 GB machine, the official advice is to download DeepSeek V4 Flash Q2. Downloads go into gguf/, and rerunning the same command resumes an interrupted download.

Once a model is downloaded, everyday use looks like this. The default model is ds4flash.gguf, a link updated by each main-model download; pass -m FILE to choose explicitly. The server listens on http://127.0.0.1:8000 by default.

The interactive CLI supports /help, /read FILE, /ctx N and /quit, and Ctrl+C interrupts generation. Thinking is on by default; --nothink gives direct answers. For V4.1, --think-level 25 sets reasoning effort from 1 to 100 (0 disables thinking). Sampling defaults are temperature 1, top-p 1, and min-p 0.05, and --temp 0 selects greedy output.

Resident inference is faster whenever the model fits; SSD streaming trades speed for capacity. It keeps a bounded cache of routed experts in memory and reads missing ones from the GGUF, but it does not remove the memory needed for other weights, activations, scratch, and context. The official advice is to start with the automatic budget; the startup report shows the effective cache and memory requirements.

Reference numbers on a 128 GB M5 Max (September 6, 2026; automatic cache, no speculative decoding): GLM 5.3 Flash Q4_K (177.77 GiB) gave 121 t/s initial prefill, 104 t/s continued prefill, and 11.9 / 14.9 t/s generation (three-run median); DeepSeek Flash Vision Exp MXFP4 (145.26 GiB) gave 300 t/s, 263 t/s, and 11.9 / 19.3 t/s (single run). Generation is more sensitive to cache misses than prefill, so a model that starts can still be too slow for interactive work; the docs recommend a short test generation first. PRO Q2 on a 128 GB Mac is described as usable for inspection but slow. These are workload references, not guarantees.

One more trap: an oversized expert cache can push out the non-routed weights that every token needs and slow decoding. More cache only helps while the rest of the working set still fits.

Through ds4-server, ds4 can back coding agents such as Pi, OpenCode, Codex CLI, and Claude Code. Start the server first and keep the client's context limit at or below the server's; output tokens also consume context, and a client setting does not enlarge the server allocation. The dsv4-local strings below are placeholders, not authentication.

Claude Code uses the Anthropic-compatible endpoint. The official shell wrapper points the main agent, the model roles, and subagents at the local DeepSeek V4 Flash. Give the wrapper a name different from claude.

Codex CLI uses the Responses API: add a provider to its configuration and launch it as shown. OpenCode takes an @ai-sdk/openai-compatible provider in ~/.config/opencode/opencode.json (baseURL http://127.0.0.1:8000/v1), with ds4/deepseek-v4-flash selected as the model. Pi works similarly through ~/.pi/agent/models.json.

Agent clients send large initial prompts, so the first prefill can take a while. Enabling the disk cache with --kv-disk-dir lets later sessions reuse compatible prefixes.

These are the recorded DeepSeek V4 Flash Q2 baselines from the official PERFORMANCE.md, using 2048-token continued-prefill intervals and 128 greedy generation tokens at each frontier. They are baselines, not fresh measurements of every commit.

The pattern: the M5 Max generates faster but its prefill drops as context grows, while the DGX Spark holds prefill in the 800s even at long contexts but generates at less than half the M5 Max's rate. Which one suits you depends on whether you feed long inputs or want snappy replies. For more on running DGX Spark locally, see confidential code analysis on DGX Spark. To measure your own machine, the bundled ds4-bench reports prefill and generation per context frontier.

Speculative decoding is opt-in. GLM and Qwen use --mtp with a built-in draft block, so no second model file is needed; V4 Flash DSpark needs a matching support GGUF. It can improve generation, but not every workload benefits, and on unpredictable prompts it can be slower than plain decoding. See the speculative decoding doc for the difference between default opportunistic sampling and --mtp-exact-sampling.

With two 128 GB Macs connected by RDMA, you can run 4-bit DeepSeek Flash or GLM 5.3 Flash with tensor parallelism. For V4.1 Flash Q2, each rank holds about 81 GiB of main weights and both machines need the full GGUF on disk. V4.1 Q4 does not fit resident TP on two 128 GB Macs, so it needs streaming or a 512 GB Mac. Pipeline parallelism can also pool the RAM of several systems.

The core difference is a general-purpose runner versus a narrow engine specialized for a few models. ds4 builds on llama.cpp's work, so it is less a competitor than a different tool for a specific job. The comparison below sticks to what the docs support.

If you swap models often for everyday work, general tools like those in our Ollama vs LM Studio comparison are easier to live with. If you want continuous batching and menu-bar management on Apple Silicon, see our oMLX guide. ds4 is for people who value running DeepSeek V4 or GLM 5.x-class MoE models realistically on a 128 GB-class machine they already own.

- Beta quality: it changes very quickly; a QA run precedes each release, but instabilities and regressions are possible
- Project GGUFs only: other GGUFs may have unsupported tensor layouts, metadata, or quantization mixes; use the download_model.sh targets
- Built with strong AI assistance: the author states openly that it was developed with strong assistance from AI coding agents, with humans leading ideas, testing, and debugging, and says that if you dislike AI-developed code, it is not for you
- A fast local SSD is assumed: V4.1 Engram tables and Qwen3.8 n-grams are always read from disk, and SSD streaming makes SSD speed decisive
- Uneven feature support: V4.1 vision is Metal-only, agent sessions with images cannot be saved with /save, and GLM requires --power 100 and rejects --prefill-chunk and an external --mtp-model
- Privacy: saved conversations and traces may contain private information; point clients only at servers you trust
- No bit-exact guarantee: scalar, batched, and tensor-parallel execution are not numerically identical

The author also expects users to adapt the software with coding agents for their own setups. The official stance is that things not documented or implemented may be easy to achieve by asking an agent.

DwarfStar (ds4) is an MIT-licensed, purpose-built inference engine from the creator of Redis, built around running a few excellent models on hardware you can own. By combining asymmetric 2-bit quantization, SSD streaming, disk KV caches, MTP, and multi-machine setups, it aims to make large models such as DeepSeek V4 and GLM 5.x realistic on 128 GB-class Macs and the DGX Spark, and ds4-server lets it back Claude Code and Codex CLI. It is also beta, runs only project-made GGUFs, and depends on SSD speed. A practical rule: pick established general tools for versatility and stability, and consider ds4 when you specifically want to run those large models locally.