Needle 2: Cactus Compute's 14MB On-Device Agent LLM

Needle 2 is an ultra-small on-device agent LLM built by Cactus Compute. It has 45M parameters and, after 2-bit quantization via Cactus Quants (CQ2), ships as a single 14MB binary with the model weights baked directly into the inference engine — there's no separate model file to download. Peak RAM usage per session is roughly 28MB. Rather than being a general-purpose chatbot, Needle 2 is purpose-built for three tasks: tool calling, device use, and structured extraction — an "agent-only micro model." It's released under the Apache 2.0 license on GitHub (cactus-compute/needle) and Hugging Face (Cactus-Compute/needle2), and made waves after a Show HN post.

The unusually short 256-token context window reflects the model's narrow focus: single-turn tool-calling decisions rather than open-ended conversation. That design choice is what keeps both the model size and RAM footprint this small.

Needle 2's capabilities fall into three buckets. First, tool calling: given a registered set of functions, it picks the right function and arguments based on the user's utterance, with a built-in retrieval step that selects the top 5 tools per turn even from a large tool catalog. Second, device use: it can generate operation commands for phones and IoT devices. Third, structured extraction: by pairing the model with a Pydantic model, output types are strictly specified, and byte-level grammar constraints guarantee the output conforms to the defined schema.

Responses also come with a confidence score in a "confidence-gated" design — when confidence is low, the system can escalate to a larger cloud model instead. That lets simple tasks resolve entirely on-device while only the hard calls get routed to the cloud, forming a practical hybrid setup. The underlying tool-calling approach shares a lot of common ground with the local inference engines compared in ../columns/local-llm-inference-engine-comparison-2026, which is worth reading alongside this piece for a fuller picture of agent operations.

These numbers are simply out of reach for typical 7B–70B LLMs. It's the combination of a 45M-parameter model and 2-bit quantization that makes real-time tool calling possible on modest CPUs and low-power SoCs.

You can install Needle 2 from Python with pip and try it right away.

Registering a tool is as simple as decorating a Python function.

For structured extraction, define the data you want to extract as a Pydantic model and pass that type to Needle 2, and it returns type-safe, schema-conformant output. Fine-tuning on Mac or PC against your own custom tool schema is also supported.

Needle 2 ships as a dependency-free single C++ binary. It runs at practical speeds not just on single-board computers like the Raspberry Pi 5, but also on sub-$200 budget phones (Samsung A-series, etc.). It also has static library support for Cortex-M microcontrollers, putting embedded devices and IoT on-device AI within reach. With peak RAM around just 28MB, it can stay resident on memory-constrained edge devices without crowding out other processes. Note that dedicated SDKs for Flutter, React Native, Kotlin, and Swift are not documented on the official page — for now, integration is fundamentally C++ binary based. For comparison against larger local LLMs and their hardware requirements, see ../columns/qwen3-8-27b-requirements-vram-local-2026 and ../columns/nemotron-3-requirements-vram-local-2026, which help clarify the right model size for a given use case.

According to the information confirmed so far, Needle 2 trades wins and losses roughly evenly against FunctionGemma 270M, LFM2.5 230M, and Apple FM, while being 5–70x smaller in model size — comparing Needle 2's 2-bit quantized footprint against the other models' f16 representations. In other words, its main differentiator is delivering comparable performance in a dramatically smaller footprint. Liquid AI's LFM2.5 is also covered in ../columns/liquid-ai-lfm25-japanese-models-2026-06, useful as a reference point when comparing small models including Japanese-language support.

Needle 2 is well suited to tasks that repeat a narrow, bounded decision at high speed and low memory: selecting and executing from a predefined set of functions, generating operation commands for phones or IoT devices, or extracting data into a fixed schema. It also works well as an orchestration layer in an agent pipeline, letting Needle 2 handle simple decisions on-device while escalating only the hard cases to a cloud LLM.

On the other hand, the 45M-parameter size and 256-token context window make it a poor fit for long-form summarization, general chit-chat, or tasks requiring complex reasoning. It's a small, specialized decision engine for tool calling and structured extraction — not a substitute for a general-purpose chatbot, and that limitation is worth being upfront about.