NVIDIA Open-Sources 74B Agent Model That Runs on 24GB of VRAM

While the Whole Industry Tightens Up, NVIDIA Chooses Openness

Over the past month, major players have moved almost in lockstep toward closed-source approaches. Meta's Muse Wonder is offered via API, OpenAI's GPT-6 is locked inside ChatGPT, and Google's Gemini 4 Argon likewise only exposes an interface. Users pay by the token, and data must leave local devices.

NVIDIA has taken the opposite path. Its released Nemotron 3.5 Bolt 74B-A9B is a 74-billion-parameter agent model that the company says can run in 24GB of memory, with the model weights, training data, and training recipe all made public. This is not a demo version, nor a preview requiring approval—it's the entire model, available for download.

One honest caveat is in order: as of publication, whether the weight files are actually live has not been confirmed, and all current information comes from pre-release coverage. It's worth checking Hugging Face or Ollama directly to confirm actual download availability rather than assuming it's already in place.

The Technical Logic Behind "74B Squeezed into 24GB"

The model name itself is a miniature spec sheet. 74B is the total parameter count, A9B means roughly 9 billion parameters are activated per token, and 24GB is the claimed footprint of the compressed weights.

Bolt uses a sparse Mixture-of-Experts (MoE) architecture. Only about 9 billion parameters participate in computation per token, so the per-token compute looks like that of a 9B model, but the stored knowledge spans 74B. The key point: inactive parameters still need to be stored. MoE saves compute, not storage.

The real storage trick comes from 2-bit quantization combined with quantization-aware training—parameters are stored at 2 bits instead of the usual 16 bits, and the model anticipates this compression during training rather than having it forced on afterward. This is the core reason 74 billion parameters can be compressed to around 24GB.

But the details deserve attention. 24GB is only NVIDIA's claim for the weights themselves, not the full system requirement. Context length, KV cache, runtime overhead, and agent software all consume additional memory. Treat 24GB as a floor, not a comfortable target.

Sparse Mixture-of-Experts (MoE) architecture works by splitting a large feed-forward network into many "expert" sub-networks, then using a lightweight routing network to decide which few experts each token should be handed to. In a traditional dense model, every token activates all parameters; MoE activates only the selected few experts, so the total parameter count can be far larger than the per-computation amount. This is where the name 74B-A9B comes from: 74B is the sum of all experts in the repository, and A9B is the roughly 9 billion parameters actually triggered per token.

Two concepts need to be distinguished: activated parameters determine the compute cost at inference time, while total parameters determine the volume of weights that must reside in memory. MoE reduces the former; the latter does not automatically shrink just because the model is sparse. In other words, MoE makes a model "fast to compute," but not "small to store." To truly fit 74B into 24GB, you must also rely on quantization to compress each parameter's bit width from 16 bits down to 2. Only the combination of the two makes NVIDIA's approach runnable on consumer hardware.

Quantization-aware training (QAT) means simulating low-precision arithmetic during the training phase, letting model weights adapt to the errors introduced by quantization, rather than doing a one-time compression after training ends. This "train first, quantize later" approach is called post-training quantization (PTQ), and it typically preserves quality at 8-bit or even 4-bit, but near 2-bit the errors grow noticeably.

The cost of QAT is needing control over the full training pipeline and investing in recomputation; the benefit is that degradation at extremely low bit widths is more controllable. NVIDIA also released the training data and recipe, in theory letting outside researchers reproduce and audit exactly how much capability the 2-bit compression sacrifices. That said, lower bit widths generally place higher demands on inference kernels—whether consumer-grade frameworks can efficiently execute 2-bit matrix multiplication directly affects real-world speed and memory usage, and this needs hands-on testing rather than relying on marketing numbers.

Open Source vs. Closed Source: Two Fundamentally Different AI Philosophies

The core difference between Bolt and this month's other releases is openness. Muse Wonder keeps its weights closed and charges $4/$20 per million tokens; GPT-6 Sol is closed and charges $2/$10; Gemini 4 Argon is likewise closed, offered at an entry price of $2/$10. Bolt, by contrast, is free to download—you only pay for hardware and electricity.

The practical impact is concrete. Open weights mean you can fine-tune the model for your own agent scenarios instead of doing prompt engineering around someone else's API. It can run fully offline, which is crucial for privacy-sensitive work and scenarios where data cannot leave local devices. Researchers can inspect the training data and recipe rather than merely calling a black box.

But open does not mean equal. Closed models still lead on independent benchmarks, and Bolt currently has no independent evaluations. NVIDIA claims it is "faster than dense models and smarter than MoE models of comparable footprint"—which amounts to NVIDIA grading its own homework. Until independent benchmarks arrive, such claims should not be treated as verified fact.

Open weights and open source are not equivalent in the AI context. Open weights means the model parameter files can be freely downloaded, run, and (within license limits) fine-tuned; open source usually also requires releasing training code, data, and the full recipe so that third parties can reproduce it from scratch. NVIDIA has released the weights, training data, and training recipe together, approaching the latter, but how far it ultimately goes still depends on the formal license terms—some "open" licenses restrict commercial use, redistribution, or derivative models.

The practical significance for users lies in control: with weights in your own hands, the model won't break because of an API redesign, price increase, or shutdown, and data doesn't have to leave local devices. The trade-off is that you bear the compute, deployment, evaluation, and quality control yourself, and once upstream updates stop, capability gradually falls behind closed-source frontier models that keep iterating.

How to Actually Run It, and the Ecosystem Layout

At launch, NVIDIA listed five runtimes: llama.cpp, vLLM, Ollama, Unsloth, and LM Studio, plus two agent frameworks, Hermes and OpenClaw^Ad.

The simplest route is via Ollama or LM Studio, pulling and running directly with the official model name. Tinkerers can choose llama.cpp (which is also the underlying engine of NVIDIA's newly launched free Llama for Windows app). vLLM serves the model as an API endpoint; Unsloth is for fine-tuning the model on your own data.

Two warnings before downloading. First, be sure to verify the exact model name on official channels—erroneous or counterfeit listings appear quickly around popular releases, and guessing the model ID could pull an unofficial version. Second, confirm the license before building a business on it; the weights were not public at release, and the exact terms must wait until the official launch.

Bolt doesn't arrive alone. It is the software flagship for the RTX Spark laptop line—which supports up to 128GB of unified memory, with retail pricing not yet announced by NVIDIA. Also released was PAIR, a free and open-source Personal AI Router graduating from beta.

PAIR distributes inference tasks across PCs in a home network. One commonly misunderstood detail: it routes an entire task to a single machine rather than pooling GPU memory. Three 24GB PCs do not merge into one 72GB virtual GPU. What you get is parallel throughput—in NVIDIA's own demo, a five-agent inbox task took 18 minutes on one machine and dropped to under 9 minutes spread across three.

The runtimes mentioned here each have their own roles, and it's worth distinguishing them before choosing. llama.cpp is a lower-level C/C++ inference engine, strong at running quantized models on CPU, consumer GPUs, and hybrid setups; Ollama and LM Studio sit on top of it, offering friendlier model management and graphical interfaces for quick experimentation; vLLM targets the server side, known for continuous batching and high throughput, and is often used to wrap models as OpenAI-compatible API endpoints; Unsloth focuses on fine-tuning, optimizing memory and speed so personal hardware can handle LoRA-style training.

The quantization formats and kernel implementations behind them are not entirely consistent, and the same model can have different memory usage and generation speeds across runtimes. So when you see "runs in 24GB," you should also confirm which runtime and what context length it was measured with. Additionally, the KV cache grows linearly with context length, and in long-context scenarios it can consume more memory than the weights themselves—an important reason to treat 24GB as a floor rather than a comfortable value.

Verdict and Recommendations

Bolt is the most noteworthy AI release of the month precisely because it rejects the month's dominant pattern. But verification is still ongoing: the weights were not confirmed live at publication, there are no independent benchmarks, and the real quality impact of aggressive 2-bit quantization will only be revealed by hands-on testing. These are not reasons to dismiss it, but reminders to test it rather than trust it.

If you want a privacy-preserving, offline-capable, fine-tunable agent model that runs entirely on your own hardware, this is worth watching. If you need maximum raw capability and are willing to pay API fees, frontier APIs remain the answer. The hybrid approach—Bolt for everyday tasks, frontier APIs for the hardest 10%—is currently the most pragmatic default for production environments.

One-line Summary

NVIDIA bucks the trend by open-sourcing the 74B agent model Nemotron 3.5 Bolt, with weights compressible to about 24GB.

Paragraph Summary

In a month when Meta, OpenAI, and Google collectively took the closed-source API route, NVIDIA went the other way with Nemotron 3.5 Bolt 74B-A9B: a sparse mixture-of-experts agent model with 74B total parameters that activates only about 9B per token, and which NVIDIA says compresses to roughly 24GB after 2-bit quantization-aware training, with weights, training data, and training recipe all released. The article explains the technical logic behind that number, noting that MoE saves compute rather than storage and that the real compression comes from quantization; it also cautions that 24GB is only the weight floor, excluding context, KV cache, and runtime overhead. It compares the closed-source camp's pricing, lists five runtimes and the PAIR inference router, and stresses that the weights are not yet confirmed live and lack independent benchmarks, advising users to test before trusting—use Bolt for everyday tasks and leave the hardest jobs to frontier APIs.