Running a 120B Nemotron on an 8GB GPU: Skip the Zeros, Find the Real Bottleneck

Title

Running a 120B Nemotron on an 8GB GPU: Skip the Zeros, Find the Real Bottleneck

Meta Description

A developer runs the 120B Nemotron-3.5-Super MoE model on an 8GB RTX 5060 at 8.2 tok/s. Key insight: measure bytes per token, skip ReLU² zeros, and discover the real bottleneck is NVMe reads, not Python overhead.

Article Content

Running a 120B-parameter large model on 8GB of VRAM? NVIDIA's Nemotron-3.5-Super says yes—but only if you change your approach: stop fixating on the model's total size and instead calculate how many expert weights need to be read per token. One developer (working with Claude) achieved this on an RTX 5060 and discovered a real bottleneck that wasn't what anyone expected.

The Core Idea: Bytes Per Token Is What Matters

Nemotron-3.5-Super is a mixture-of-experts (MoE) model with a total size of 75GB (NVFP4 format), but each token activates only 22 of 512 experts (top-22). The key is that NVIDIA made the experts very small: they operate in a 1024-dimensional "latent" space rather than the full 4096 dimensions. A single expert (up 2688×1024 + down 1024×2688) is only 3.10MB in NVFP4.

Do the math: 22 experts × 40 layers × 3.10MB ≈ 2.7GB per token. That number sits between Qwen (1.3GB/token) and DeepSeek^Ad (4.2GB/token). Meanwhile, the non-routed portion—Mamba, attention, shared experts, routers, and so on—totals 6.6GiB and stays resident on the GPU. The 59GiB of routed experts live on the NVMe drive and are streamed in on demand.

NVFP4 is a 4-bit floating-point format introduced by NVIDIA on the Blackwell architecture, designed specifically for low-precision inference. Unlike traditional INT4, it preserves a floating-point representation with 1 sign bit, 2 exponent bits, and 1 mantissa bit, and uses two-level scaling: every 16 consecutive values share an FP8 block scaling factor, plus a global scaling factor. This design maintains good numerical stability at extremely low precision, allowing 120B-class model weights to be compressed to around 75GB. The "every 16 consecutive values share a scale" mentioned in the article comes from this—which is also why you can't simply skip zeros on a per-neuron basis: breaking the 16-value grouping would lose the scaling information. NVFP4's hardware decompression path requires Blackwell or newer GPUs to execute efficiently, which is one of the reasons the author emphasizes the need for a Blackwell GPU.

Mixture-of-experts (MoE) models are built on the idea of splitting the feed-forward network into multiple "experts," with each token selectively activating only a few of them via a router. Unlike dense models, where every token passes through all parameters, MoE's activated parameter count is far smaller than the total, keeping inference compute manageable. But MoE's VRAM challenge is this: even though compute only involves a few experts, the weights of all experts still need to be stored. Nemotron-3.5-Super uses top-22 routing, activating 22 of 512 experts per token, and the experts operate in a 1024-dimensional "latent" space, lower than the full 4096 dimensions, making a single expert just 3.10MB. This "small experts, many of them" design reduces the bytes of experts that must be read per token—and is the prerequisite for running a 120B model on 8GB of VRAM.

Breaking Down the Engine

With no off-the-shelf runtime available, the author built an engine on PyTorch + Triton, using transformers' NemotronH code as the skeleton. A few key design decisions:

1. Disk block design: Each expert's four tensors are rewritten into a single 3,096,576-byte block—exactly 756 4KiB pages. One pread with O_DIRECT reads it into pinned memory. Repacking 59GiB takes about 2.5 minutes.

2. RAM cache: LRU strategy with 32 read threads. Routing is highly concentrated—with 24% of experts in RAM, 81-84% of lookups hit.

3. Prefetching: Run the next layer's router on the current input, which guesses correctly about 72% of the time. Misses are read while the GPU is busy.

4. Skipping zeros (the new trick): After ReLU², 92.5% of an expert's neurons are zero for a given token. Zero neurons don't need their down columns, so they don't have to be moved over PCIe. But NVFP4 shares a scale across every 16 consecutive values, so you can't just pick individual neurons. The author re-laid out the down weights on disk (a pure byte rearrangement) so that each neuron's 4-bit values are contiguous and scales are grouped by 16 neurons. Now only the up (1.55MB) is copied to the GPU; the down kernel reads directly from pinned memory, and only the 16-neuron groups containing active neurons. This gave a 12% speedup: 6.6 → 7.3-7.6 tok/s.

5. Chat server: An OpenAI-compatible API with streaming support and conversation snapshots (~170MB) so each new message only processes the new tail.

O_DIRECT is a Linux file I/O flag that bypasses the kernel page cache, transferring data directly from the storage device to a user-space buffer. When streaming expert weights, using O_DIRECT avoids wasting precious memory on double caching (kernel page cache + user-space cache) and reduces CPU overhead. However, it requires that read offsets and lengths be aligned to the device block size (typically 4KiB), and the buffer address must be aligned too. The author rewrites each expert into a 3,096,576-byte block—exactly 756 4KiB pages—precisely to satisfy O_DIRECT's alignment requirements and complete an entire expert read with a single pread. Combined with pinned memory, this also lets the GPU asynchronously copy directly from host memory over PCIe, reducing transfer latency.

The Real Bottleneck: Disk Reads, Not Python Overhead

At 7.3 tok/s, each token takes about 137ms. The author assumed this was Python launch overhead and planned to optimize with CUDA graphs. But Nsight Systems profiling showed: the GPU is busy for about 92ms and idle for about 54ms; of that, the gap between launches is only about 13ms. CUDA graphs could only address those 13ms.

Two "obvious" fixes were tried—reordering the work so shared experts run before routing results, and copying predicted experts early—both bit-for-bit correct, but neither helped. In fact, speculative copying was 7% slower because wrong guesses clogged the bus.

The actual wait times: NVMe reads (RAM cache misses) account for 50-55ms, Python cache bookkeeping 17ms, and waiting for routing decisions 11ms. Disk is the biggest wait by far. The earlier measurement that said "disk ~3ms" was wrong—that run had GPU copying disabled, which masked the disk latency.

The fix was mundane: increase the RAM cache. Simulating routing offline and choosing a 20GB cache cut disk reads from 136.9ms to 121.5ms and raised speed from 7.3 to 8.2 tok/s. That leaves about 7GB of RAM for the rest of a 32GB system.

Vision Capabilities and Correctness Verification

Nemotron-3.5-Super-VL comes with a 1.3GB vision tower (C-RADIO). The engine's 6.6GiB plus the vision tower won't fit on the GPU simultaneously, so the vision tower's weights are placed in pinned memory and streamed layer by layer to the GPU (0.4-1.6 seconds per image, versus 10-31 seconds on CPU).

Tested on an invoice with 60 lines in a 15px font: at normal resolution, 7-9 out of 10 fields were correct; at NVIDIA's maximum resolution (2204 tokens), all three fields scored 10/10, taking 116 seconds total.

For correctness, the author used an fp32 transformers implementation on CPU as the reference, with the original checkpoint (not the repacked version) and transformers' own FP4 decoding. Perplexity (PPL) comparison: English passage 5.233 vs 5.280, TCP vs UDP Q&A 6.592 vs 6.714, Russian Q&A 5.873 vs 5.854. Average KL ≤ 0.003. Greedy answer match: 63/64 and 64/64, with the only difference being a genuine tie (the reference itself had a 0.11 token score gap, smaller than one BF16 step).

They also ran a check that had to fail: swapping the two 4-bit halves of every byte sent PPL skyrocketing to 18,784—proving the decoding order is actually tested rather than assumed.

Perplexity (PPL) is a metric for a language model's predictive ability, defined as the exponent of the model's average negative log-likelihood per token over test text. Lower PPL means the model predicts the text more accurately. When validating quantization or custom inference engines, PPL from the optimized implementation is typically compared against a reference implementation (such as fp32 transformers on CPU); if the difference is small (like 5.233 vs 5.280 here), the numerical error is within acceptable bounds. KL divergence measures the difference between two probability distributions; an average KL ≤ 0.003 here indicates the custom engine's output token probability distribution is highly consistent with the reference. Greedy answer matching further checks whether token-by-token generation matches; 63/64 and 64/64 match rates show differences occur only where the reference itself has near-tie probabilities.

What Didn't Work

• Putting the prompt's experts into the RAM cache—made prompts slower with no decoding benefit.
• Prefetching 6 guesses instead of 4—more wasted reads.
• Skipping zeros on a per-neuron basis—re-reading scales neuron by neuron was slower than copying the whole block. Grouping by 16 neurons worked.
• Speculative early copying—correct but slower.
• Keeping recently used experts on the GPU—40% repeat on the next token, but storing them needs about 2.7GB of VRAM, and an 8GB card has no room next to the 6.6GB dense portion.

How to Try It

The repository includes the engine, Triton kernels, server, repacking/verification tools, and all measurement scripts. An honest caveat: it hasn't been run end-to-end from a fresh clone. You need a Blackwell GPU (the kernels use NVFP4 hardware paths), about 70GB of space on fast NVMe, and 32GB of RAM.

The most important takeaway: for MoE models, "does it fit in VRAM" is the wrong question. Calculate bytes per token, then measure where the time actually goes before fixing it—because the obvious culprit (Python launch overhead, in this case) may be wrong.

One-line Summary

Using NVFP4 and a bytes-per-token approach, a developer runs a 120B MoE model at 8.2 tok/s on an 8GB GPU—and finds disk reads are the real bottleneck.

Paragraph Summary

A developer working with Claude runs the 120B-parameter Nemotron-3.5-Super MoE model on an RTX 5060 (8GB VRAM). The key idea is to abandon the old question of "does the model fit in VRAM" and instead calculate the expert bytes to read per token: the model activates only 22 experts per token, about 2.7GB/token, with 6.6GiB of non-routed parts resident on the GPU and 59GiB of routed experts streamed from NVMe on demand. The engine is built on PyTorch + Triton with O_DIRECT disk blocks, an LRU RAM cache, and next-layer router prefetching, and exploits the fact that 92.5% of neurons are zero after ReLU² by re-laying out down weights to transfer only active neuron groups—a 12% speedup. Nsight analysis reveals the real bottleneck is NVMe reads (50-55ms/token), not Python overhead; increasing the RAM cache to 20GB raised speed from 7.3 to 8.2 tok/s. Correctness is validated via PPL and greedy matching, with average KL ≤ 0.003.