WASTE: Run Kimi K3 (2.78T) on 29GB RAM, No GPU

A 29GB-RAM laptop can run Kimi K3's 2.78 trillion parameters. The catch is throughput of just 0.45-0.62 tok/s. WASTE (sqliteai/waste), an inference engine that hit the front page of Hacker News on August 1, 2026 (328 points), combines the sparse activation of MoE models with NVMe streaming to run models that would normally demand hundreds of gigabytes to terabytes of memory on an ordinary laptop. This article covers how WASTE works, the measured specs, how to install it, and how far 0.5 tok/s actually gets you.

WASTE (sqliteai/waste) is a dependency-free inference engine written in C11, needing nothing beyond libc and pthreads. It runs on macOS, Linux, and Windows (via MinGW-w64), and is Apache 2.0 licensed. The GitHub copyright is held by SQLite Cloud, Inc., and SQLite's own author has publicly acknowledged involvement with the project on Hacker News. Unlike general-purpose engines such as llama.cpp, WASTE is purpose-built around one problem: running massive MoE models without loading every weight into memory at once.

The intended targets are trillion-parameter-class Mixture-of-Experts models like Kimi K3. K3's own architecture and API pricing are covered in Kimi K3 explained: 2.8T MoE specs, pricing and local requirements, and the self-hosting math after the weight release — which concluded you need roughly 64 H100-class GPUs — is laid out in Kimi K3 open weights: what it actually takes to self-host a 2.8T MoE. WASTE effectively replaces that cluster requirement with a single laptop, for a real speed cost.

The trick lies in how MoE models work. Only about 4% of all parameters are actually activated for any given token. WASTE exploits this: the shared portion of the model (the trunk) stays resident in RAM, while the weights of whichever experts get selected per token are streamed from NVMe on demand. The on-disk layout is optimized so that reading one expert requires exactly one aligned read, avoiding wasted seeks and fragmented I/O.

- Shared trunk: only the weights used by every token stay resident in RAM
- Selective expert reads: only the experts a given token actually needs are pulled from NVMe (roughly 4% activation)
- Lookahead router: predicts which experts will be needed before the real router finalizes its decision, kicking off I/O early
- Compute/I-O overlap: thanks to the lookahead, disk reads proceed alongside compute — inference output itself is unchanged regardless of lookahead
- Spare RAM as expert cache: leftover memory serves as a bounded cache; a higher hit rate directly raises effective speed

As a result, even when total model size far exceeds available RAM, only the trunk plus a slice of active experts ever needs to sit in memory. For Kimi K3, the default 17.56GB expert cache is reported to achieve a 36.2% hit rate.

Kimi K3's asymmetry — a 982GB container against a 29.06GB minimum RAM footprint — is the clearest demonstration of what WASTE is actually for: decoupling model size from memory capacity. Kimi-Linear 48B, with its much smaller container, needs only 1.28GB of RAM and reaches double-digit throughput. The pattern is clear: the smaller the model relative to its NVMe footprint, the less streaming dominates, and the faster it feels in practice.

WASTE's performance is tightly coupled to storage speed. In published measurements, internal NVMe hit 12.78 GB/s while a tested USB external drive managed only 0.94 GB/s — roughly a 13.6x gap. Trying to run a Kimi K3-class model off external storage is unlikely to deliver usable throughput. Before setting anything up, the first thing worth checking is your machine's actual internal NVMe read speed.

The expert weights WASTE ships are compressed with 3-bit residual vector quantization, while the higher-impact shared weights are kept at 4-8 bit. Alongside NVMe streaming, this quantization scheme is the other pillar keeping RAM requirements down. On top of that, Kimi K3 itself uses linear attention with compressed latent KV, bringing the KV cache for a 4K context down to just 0.21GB (versus an estimated 11.25GB under conventional attention). That's a property of K3's architecture rather than a WASTE-specific optimization, but it substantially shrinks how much of the RAM budget the KV cache consumes — and it's directly why the 29GB floor is reachable at all.

No special environment is needed to build it — a C11 compiler and make are enough.

git clone https://github.com/sqliteai/waste
cd waste
make && make check

Once built and verified, run a single inference with ./waste run or start an interactive session with ./waste chat. An OpenAI-compatible chat-completions HTTP server ships alongside the CLI, supporting streaming, structured output, and tool use, so existing OpenAI SDK client code can largely be pointed at it unchanged. Note that converting weights (from PyTorch/safetensors format into WASTE's own format) is a one-time offline preprocessing step, not something inference itself needs — it takes roughly 4.7 hours for a Kimi K3-class model even with three parallel workers. PyTorch and safetensors are not required at inference time.

WASTE also handles image input. The CLI accepts a --image flag, and /image works mid-conversation in chat mode. An 896x896 image expands to 256 prompt positions, each taking roughly 2.8 seconds to process — meaning a single image adds close to 12 minutes of processing time on top of whatever text follows.

0.45-0.62 tok/s is not a realistic number for anything expecting an immediate response, like chat. Commenters on Hacker News pointed out that at this rate, generating one million tokens takes about 23 days — a non-starter for conversational use. The 4K context limit narrows the field further, ruling out long-context workloads.

That said, for batch processing, offline inference, or async tasks where a response minutes or tens of minutes later is fine — background document summarization, or experimentation during development — the value is simply being able to get output from a 2.78-trillion-parameter model at all, without a cluster or a cloud API. Power draw is reported at around 42W sustained; at $0.20/kWh that works out to roughly $5 per million tokens. Several Hacker News commenters noted that cloud APIs are usually cheaper for the equivalent token volume, which is a fair point — running it locally and running it cheaply are two separate questions.

One more thing worth flagging: Hacker News also criticized the README itself, calling the initial version verbose "LLM slop," and the author committed to rewriting it by hand. It's reasonable to treat the documentation as still a work in progress.

The established way to run massive MoE models under tight memory has generally been llama.cpp's mmap-based offloading or splitting layers between GPU and CPU. Both are essentially variations on "put the whole model on disk and let the OS page cache handle reads as needed," with no mechanism for predicting which expert will be needed next. WASTE's differentiator is that it designs the on-disk layout and access pattern specifically around MoE's sparse activation pattern, and starts speculative I/O before the real router even finalizes its decision. Since the build has no dependencies beyond make, there's no heavyweight CUDA/cuDNN toolchain to set up. That said, CUDA and Metal backends remain experimental for now, with CPU inference doing most of the work — and on Apple Silicon, the ARM NEON implementation reportedly outperforms Metal.