NVIDIA Releases Nemotron 3.5 Bolt: A 74B-Parameter Local VLM Squeezed into 24GB of VRAM

NVIDIA has released Nemotron 3.5 Bolt 74B-A9B 24GB on Hugging Face—a vision-language model (VLM) with 74B total parameters and 9B active parameters. Its core selling point is using native 2-bit quantization to squeeze the entire model into 24GB of VRAM, so it can run on a single consumer-grade or workstation GPU while still leaving resources for other programs.

How 74B Parameters Fit into 24GB

The answer given by the model card is a combination of techniques: architecturally, it uses LatentMoE, a hybrid of Mamba-2 and Transformer, with a C-RADIO v4-H encoder on the vision side; for precision, it goes straight to IQ2, meaning 2-bit quantization with quantization-aware training (QAT). NVIDIA calls this "native" quantization rather than post-hoc compression, with the goal of optimizing the intelligence and speed obtained per GB of VRAM.

The parameter structure is 74B total with 9B active. This "large body, small activation" design keeps the actual compute during inference far from the 74B figure, and together with 2-bit weights it ultimately packages into a 24GB deployment footprint. Context length is supported up to 256K tokens, input supports text, images, and video, and output is text.

MoE and active parameters: Mixture-of-Experts (MoE) splits what would originally be a single feed-forward network into multiple parallel "expert" subnetworks, routing to and activating only a few of them in each forward pass. As a result, the total parameter count can be very large while the per-pass compute remains small. The 9B in "74B-A9B" refers to the scale of active parameters actually involved in computation for a single token. LatentMoE is NVIDIA's variant on this idea: routing happens in a low-dimensional latent space rather than the full hidden dimension, further reducing communication and compute overhead. This is also the prerequisite that allows it to fit a larger "body" into the same amount of VRAM.

Quantization-aware training (QAT): Ordinary post-training quantization (PTQ) directly truncates already-trained weights into low-bit representations, often causing noticeable accuracy loss. QAT instead simulates quantization error during the training stage, letting the model adapt itself to low bit widths, which is why an extreme compression like 2-bit becomes usable. "IQ2" is a 2-bit quantization format in the llama.cpp/GGUF ecosystem, using importance weighting and mixed precision to preserve the precision of key weights as much as possible at extremely low bit widths.

The Tradeoff Logic Between Speed and Intelligence

NVIDIA describes the model's positioning very directly: faster than most mainstream dense models, while smarter than mainstream MoE models with similar VRAM usage. The implied reference point is that within the 24GB VRAM tier, you can usually only run dense models with much smaller parameter counts, or MoE models whose activation efficiency is not as extreme. Bolt attempts to achieve a higher "intelligence-to-price ratio" within this constraint.

This tradeoff is critical for local agentic tasks. Everyday automation tasks often require the model to stay online continuously and be called repeatedly, making latency and VRAM usage more important than the ultimate quality of a single response.

Inference Stack and Deployment Details

On the deployment side, the model runs on vLLM and supports NVIDIA Ampere (A100), Hopper (H100-80GB), and Blackwell platforms. Worth calling out separately is speculative decoding: a companion LiLiCorr drafter is provided, implemented through DFlash, and Bolt's vLLM configuration includes 7 speculative tokens by default.

The model card also notes how to switch inference modes—following the Super 3.5 VL client's chat-template control, thinking mode is toggled via enable_thinking=True/False. The recommended sampling parameters are temperature 1.0 and top_p 0.95, also inherited from Super 3.5 VL.

In addition to the main repository on Hugging Face, there are also GGUF versions and LiLiCorr-GGUF versions, making it easier to integrate with different inference backends.

Speculative decoding: When a large model generates token by token, it must run a full forward pass at each step, and the bottleneck is often VRAM bandwidth rather than compute. Speculative decoding lets a very small "draft model" first guess several tokens in succession, after which the target model verifies them all in parallel at once; accepted tokens are adopted directly. This merges multiple serial forward passes into one, significantly improving throughput without changing the output distribution. The "7 speculative tokens" mentioned here refers to the draft length per round, the LiLiCorr drafter is the companion draft model, and DFlash is its implementation method.

vLLM: A mainstream high-throughput LLM inference engine whose core is PagedAttention—managing the KV Cache like operating system paging, reducing VRAM fragmentation and waste, and thereby supporting greater concurrency and longer context within the same VRAM. This is also an important support for making a 256K context feasible on a 24GB card.

License and Data Timeliness

The license is OpenMDW-1.1, allowing both commercial and non-commercial use. The model developer is NVIDIA Corporation, the model date range is listed as December 2025 to May 2026, and post-training data extends to May 2026.

The Nemotron series itself is positioned as "open weights, open training data, open recipe," aimed at scenarios involving the construction of specialized AI agents. Bolt is the branch of this family oriented toward "small VRAM, always-on capability."

Open weights ≠ open source: The OpenMDW-1.1 adopted by Nemotron is an "open weights" license, meaning the trained parameters are published and both commercial and non-commercial use are permitted; "open training data, open recipe" goes further, disclosing data sources and training methods so that downstream users can reproduce or even modify them themselves. This distinction is crucial for agent developers: open weights mean private deployment and fine-tuning are possible, but whether commercial use is allowed and whether attribution is required still depends on the specific license terms.

One-Sentence Summary

Nemotron 3.5 Bolt compresses a 74B MoE architecture, vision capabilities, and 256K context into 24GB of VRAM through 2-bit QAT, paired with speculative decoding and a vLLM deployment path, pointing toward locally resident everyday agentic workflows rather than a replacement for cloud-based large models.

One-line Summary

NVIDIA Nemotron 3.5 Bolt uses 2-bit QAT to squeeze a 74B, 9B-active VLM into 24GB of VRAM, targeting local always-on agents.

Paragraph Summary

NVIDIA released Nemotron 3.5 Bolt 74B-A9B 24GB on Hugging Face, a vision-language model with 74B total parameters and 9B active. It uses a LatentMoE architecture combining Mamba-2 and Transformer, a C-RADIO v4-H vision encoder, and native IQ2 2-bit quantization with quantization-aware training to fit the entire model into 24GB of VRAM, runnable on a single consumer or workstation card while leaving headroom. Context reaches up to 256K tokens and supports text, image, and video input with text output. Deployment uses vLLM, supports A100/H100-80GB/Blackwell, includes a LiLiCorr drafter for 7-token speculative decoding, and can toggle thinking mode via chat-template. The official positioning is faster than mainstream dense models and smarter than MoE models at the same VRAM, aimed at locally resident everyday agentic workflows. The license is OpenMDW-1.1, allowing commercial and non-commercial use.