Nemotron 3.5 Bolt Requirements

NVIDIA Nemotron 3.5 Bolt 74B-A9B (the 24GB build) is an open-weight MoE vision-language model with 74B total and 9B active parameters. The GGUF file is about 22GB and you need roughly 24-32GB of memory to run it. An RTX 5090 (32GB) or a Mac with 32GB of unified memory is the realistic floor; 24GB GPUs (RTX 4090, RTX 3090) need a short context or partial CPU offload (our estimate). NVIDIA released it on October 15, 2026 under OpenMDW-1.1, and the model card says it is ready for commercial and non-commercial use.

The headline is that a 74B model shrinks to about 22GB through 2-bit-class quantization plus quantization-aware distillation (QAD). Most speed and benchmark figures, however, are vendor-reported and independent verification is still thin. This article separates vendor numbers from third-party measurements and flags where sources disagree (such as context length). The memory figures are our estimates, built from the published file sizes and third-party measurements.

If you want to check quickly whether your GPU or Mac can run a model, our VRAM calculator (free, no sign-up) estimates required VRAM from model, quantization, and context length. Bolt uses mixed precision that varies by layer, so prefer the actual file sizes in the table above over a simple parameters-times-bits estimate.

Nemotron 3.5 Bolt 74B-A9B is a vision-language model NVIDIA released on October 15, 2026. It accepts text, images, and video, and supports tool calling and hybrid reasoning (a thinking mode). It has 74B total parameters and 9B active per token. As with any MoE, memory capacity is set by total parameters while compute per token is set by active parameters.

Bolt was built by compressing its parent, Nemotron 3.5 Super VL, with a method called Star Elastic. One concrete difference is expert top-K (the number of experts chosen per token): 22 in the parent versus 9 in Bolt. Fewer selected experts means lighter compute per token.

Structurally it mixes Mamba-2 (a state-space model), attention, and MoE, which tends to keep memory growth on long inputs gentler than a pure Transformer. For related models, see our notes on Nemotron 3.5 Lightning Switchyard and the Nemotron 3 requirements article.

Naive math says 74B parameters at 2 bits is about 18.5GB. The actual file is about 22GB because not everything is 2-bit: the parts that matter most for quality are kept at higher precision. The breakdown, based on NVIDIA's description, is below.

Only the experts, which dominate capacity, are compressed hard, while the smaller, more sensitive parts stay near 8-bit. NVIDIA also says quality was recovered with quantization-aware distillation (QAD) rather than rounding after the fact. That is what sets this 24GB build apart from a plain post-hoc 2-bit quantization. How well quality holds up, though, can currently be confirmed only through vendor figures and a few third-party tests.

The files are: GGUF at about 22.0GB (20.51 GiB), the vision projector (mmproj) at about 1.73GB, the vLLM-oriented safetensors build at about 24.12GB (90 files), and the BF16 reference at about 149.8GB (63 shards). The LiLiCorr speculative-decoding drafter has 1.13B parameters, a GGUF of roughly 0.93-1GB, and adds about 1-2GB of memory at runtime. NVIDIA reports acceptance lengths of 4.68 (greedy) and 4.25 (sampling).

Sources disagree on context length. The model card says 256K (262,144 tokens in the vLLM recipe), while Unsloth's guide lists 128K. NVIDIA's 24GB target assumes batch size 1 at 128K context. If you need long inputs, check the latest runtime docs and the model card.

For memory, Unsloth estimates a minimum of 24-32GB RAM for short context and 48-64GB for headroom, with more needed for long context, images, or video. In an independent test on a DGX Spark (128GB) at 262K context, Classmethod measured about 27.9 GiB resident without the drafter and about 31.4 GiB with it. The small overhead beyond the ~22GB of weights is probably because Mamba layers keep fixed-size state, so the KV cache grows less (our inference).

In practice, start at a small context such as 4K and raise it step by step if memory allows. Image and video input adds vision-encoder work and extra memory for the resulting tokens.

The vendor's 83.99 tok/s was measured on an RTX Spark, a different machine from the DGX Spark used in the independent test, so the numbers are not directly comparable. On the DGX Spark, the drafter gave about 1.56x. vLLM was slower than llama.cpp for a single request, and at concurrency 8 total throughput rises but each request drops to 13.26 tok/s. It suits one person on their own machine rather than serving many users at once.

Benchmarks are mostly vendor-reported. NVIDIA reports 50.98 on Terminal-Bench 2.1 and 66.75 on BrowseComp, and says it ranks first on PinchBench in its memory class. Artificial Analysis, a third party, gives it an Intelligence Index of 21, 4th of 67 in its class.

A dash means the item was not confirmed in our source material. Two caveats. First, Qwen3.6 35B-A3B and Gemma 4 26B-A4B replay faster in AA-AgentPerf, and Gemma 4 26B-A4B scores higher on AA-Omniscience accuracy. Second, in Artificial Analysis's testing Bolt produced 230M output tokens against a 100M median, so its thinking tends to be verbose. In Classmethod's tests, one summarization task exhausted the token budget at the default thinking setting. Compare on real wall-clock time and token use, not just rankings. For the Gemma 4 side, see our Gemma 4 hardware requirements article.

These are estimates derived from the file sizes (GGUF ~22GB plus mmproj ~1.73GB) and the measurements above.

- RTX 5090 (32GB): comfortable for text-focused use; images and video should fit if you keep the context modest
- RTX 4090 / RTX 3090 (24GB): weights plus mmproj are about 23.7GB, so even a short context is nearly full. Expect to shrink the context or offload part to the CPU; image input is hard
- Mac (unified memory): 32GB is the minimum guide; 48-64GB is more comfortable once the OS and other apps are counted. Memory bandwidth drives speed
- DGX Spark / RTX Spark (128GB): ample room for 256K-class context and the drafter, and where the independent tests were run. BF16 (149.8GB) does not fit, including for fine-tuning
- On a budget: a PC with lots of RAM and CPU offload may work, but speed drops sharply (estimate)

For an example of using a DGX Spark for local inference, see analyzing confidential code on a DGX Spark.

Supported runtimes are llama.cpp, Ollama, LM Studio, vLLM, and Unsloth. Bolt support was merged into llama.cpp master on October 15, so build from the latest master. First, download Unsloth's GGUF and the mmproj for images.

To try text only, start with a 4K context. Set MODEL_FILE to the path of the GGUF you downloaded.

For images, use llama-mtmd-cli with --mmproj and --image. For video, load a file with /video inside llama-mtmd-cli. It needs ffmpeg and ffprobe; MP4 up to 2 minutes, with limits of 1 FPS / 128 frames at 1080p or 2 FPS / 256 frames at 720p.

To run an OpenAI-compatible local server, Unsloth's command is the easy route.

For vLLM, a recipe for the quantized checkpoint on DGX Spark is published, using an fp8 KV cache and max-num-seqs 1; Ampere, Hopper, and Blackwell GPUs are listed as supported. Check the model card recipe for the exact options. Unsloth also supports fine-tuning and Unsloth Desktop. Ollama and LM Studio support the model as well, but tags can change, so search for the model name in each library to confirm.

The license is OpenMDW-1.1, and the model card says the model is ready for commercial and non-commercial use. Redistribution and derivative-model terms are set separately, so read the model card and license text as published before business use. Even with a permissive license, handling of personal or confidential data in the images and documents you feed it is a separate matter; decide storage and internal rules yourself.

- Out-of-memory (OOM) at startup: lower the context to around 4K, drop image/video input, remove the drafter to free 1-2GB, and use CPU offload if it still does not fit
- Just barely not fitting on a 24GB GPU: weights plus mmproj are about 23.7GB, so skip the mmproj for text-only use or offload some layers to the CPU
- Thinking never ends and the token budget runs out: the default thinking setting can produce very long output. Allow a large max-token limit or use a setting that reduces thinking effort
- Model fails to load or outputs gibberish: your llama.cpp build is probably too old. Rebuild from master dated October 15 or later
- Slow in vLLM: measurements show it is slower than llama.cpp for a single request; prefer llama.cpp for one user
- Video will not load: check that ffmpeg and ffprobe are installed, the MP4 is within 2 minutes, and resolution and frame limits are respected

The 24GB build of Nemotron 3.5 Bolt 74B-A9B is a 74B-total, 9B-active multimodal model of about 22GB as GGUF (about 23.7GB with the mmproj), and a 32GB GPU or Mac is the realistic floor. It is shrunk with mixed 2-bit-class precision and quantization-aware distillation, but the evidence for speed and quality leans on vendor-reported figures.

Independent measurements show about 61.6 tok/s with the drafter on a DGX Spark and a poor fit for concurrent use. Given that it is slower than Qwen3.6 35B-A3B and Gemma 4 26B-A4B in AA-AgentPerf and emits more output tokens, measure time and memory on your own workload before adopting it.