763B MoE Fits on Two 96GB GPUs: Idle Experts Offloaded to Host Memory

A single GPU has 95.6 GiB of VRAM, but after compressing the model to 3 bit, each card still needs 102.1 GiB—the gap isn't large, but it's enough to make startup die twice in a row on OOM. The core idea of the whole approach: MoE experts are idle most of the time, so don't let them occupy the card.

Where the VRAM gap comes from

DeepSeek^Ad-V4.1-Flash has 763B parameters. BF16 weights are about 1.5 TB, the official FP4/FP8 checkpoint is 475.2 GiB, and coolbho3k's 3-bit EXL3 package compresses it to 309.3 GiB. Two RTX PRO 6000 Blackwell cards add up to 192 GB, and even compressed to this degree it's still a hair short.

The key lies in the model's MoE structure: 40 layers, each with 384 routed experts, but each token only goes through 6 per layer. That is, the vast majority of experts are idle at any given moment. Diffbot's scheme moves 4,608 of the 15,360 routed experts, plus lookup tables (two Engram tables in int4, and embeddings), entirely to host memory. The GPU reads this data directly over PCIe via UVA (unified virtual addressing). Per-card usage drops to 72.4 GiB, leaving the rest for KV cache.

This number is worth remembering: if you only focus on "can the weights fit," it's easy to overlook that KV cache is the real bottleneck in multi-user scenarios.

UVA (Unified Virtual Addressing) is a mechanism provided by NVIDIA CUDA that lets the GPU and CPU share the same virtual address space. Once enabled, GPU kernels can directly dereference pointers to host memory, and data is transferred on demand over PCIe (or NVLink-C2C) without needing explicit cudaMemcpy into VRAM. This has two consequences: first, VRAM usage drops substantially because the weights physically stay in host DRAM; second, every access goes over PCIe, whose bandwidth is far lower than HBM (PCIe 5.0 x16 is about 64 GB/s unidirectional, while a single Blackwell's HBM bandwidth is on the order of TB/s). Therefore, whether UVA offload pays off depends on how frequently the offloaded data is accessed—which is exactly why MoE experts are well suited to being offloaded.

KV cache is the Key/Value tensor that caches historical tokens during LLM inference, used to avoid recomputing the entire context every time a new token is generated. Its size grows linearly with the number of concurrent requests × sequence length × number of layers. In long-context multi-user scenarios, KV cache often takes up as much or even more VRAM than the weights. Offloading weights to host memory is essentially freeing up space for KV cache, thereby supporting higher concurrency and longer contexts.

The same idea, but vLLM's native approach doesn't work

The most deceptive part is here. vLLM itself also has UVA offload, and it looks like it's doing the same thing. But after putting the same data in host memory, initialization crashes when KV block size is set to 64, and while setting it to 128 allows startup, the first request fails. A standard vLLM installation cannot serve this model on these two cards.

What does work is a combination, with a corresponding author for each layer: DeepSeek provides the weights, coolbho3k does the 3-bit quantization, turboderp wrote the EXL3 format and the ExLlamaV3 inference library, Victor Cruz wrote vllm-exl3 as the bridge layer into vLLM, Diffbot designed the per-expert split scheme, startup flow, and kernels on top of vLLM, and InferenceKey assembled these and did the measurements.

The "looks equivalent but fails" point matters for self-deployers: vLLM's native offload and the specialized per-expert plan both ostensibly put weights in host memory, but the underlying data layout and read paths are completely different.

EXL3 is a 3-bit quantization format introduced by ExLlamaV3, a successor to EXL2. It uses more aggressive bit widths and a finer quantization grid to compress weights, with the goal of fitting large models into consumer-grade or multi-GPU-constrained VRAM. ExLlamaV3 is the inference library maintained by turboderp, responsible for interpreting EXL3 weights and executing forward computation. vllm-exl3 is the bridge layer written by Victor Cruz that plugs ExLlamaV3's quantization kernels into vLLM's scheduling and PagedAttention system, enabling vLLM to load models in EXL3 format. This stack and vLLM's native quantization paths (such as AWQ, GPTQ, FP8) are two parallel implementations, with different data layouts, kernel choices, and memory management, and therefore cannot be swapped.

Per-expert plan means not treating an entire MoE layer as a single unit, but deciding, on a per-expert basis, its residency location (VRAM or host memory) and read path. Of the 384 experts per layer, only 6 are activated, and a per-expert plan allows only the frequently routed experts to stay on the GPU, pulling the rest from host memory on demand. This is finer-grained than vLLM's native "whole-layer or whole-block offload," and is one of the key differences that lets this scheme run where the native approach crashes.

PCIe topology: an easily overlooked performance variable

Once experts are placed in host memory, the path from the GPU to that memory directly determines part of the speed. On a rented machine, with the two GPUs attached to different NUMA nodes, inference speed was 1.17 to 1.51 times slower. Before getting hands-on, run this first:

nvidia-smi topo -m

**NUMA (Non-Uniform Memory Access)** describes how, in a multi-socket CPU system, each CPU socket has its own local memory, and accessing local memory is faster than accessing remote memory attached to another CPU. A GPU connects through a PCIe slot to a specific CPU's PCIe root complex, and therefore naturally belongs to a NUMA node. When two GPUs are attached to different NUMA nodes, one GPU accessing host memory on the other node requires crossing the socket interconnect (such as Intel UPI or AMD Infinity Fabric) and then going over PCIe, degrading both latency and effective bandwidth, which in measurements caused a 1.17 to 1.51x speed loss.

**`nvidia-smi topo -m`** prints a topology matrix indicating the connection type (`NODE`, `SYS`, `PHB`, `PIX`, etc.) between each pair of GPUs and between GPUs and CPU/NUMA nodes. When reading this table, `SYS` means cross-NUMA-node, and `NODE` means same-node. For inference schemes that rely on host memory offload, try to keep the NUMA node where the expert data offloaded to host memory resides on the same node as the GPU accessing it; otherwise the performance loss will be mistakenly attributed to the offload mechanism itself.
# Shows how each GPU connects to the other GPU and to the CPUs.
# GPUs on different NUMA nodes ran 1.17x to 1.51x slower here.

Otherwise it's easy to charge the cost of "the machine itself is cross-NUMA" to offload.

What should be measured is per-stream speed, not total machine throughput

What a developer feels is the speed of their own session, not the total throughput of the whole machine. So measure tokens per second per stream—one stream is one agent session. The article sets the standard at 40 tok/s per person. The median and 5th percentile data over agent turns are as follows:

Concurrent agents	Median tok/s	p5 tok/s
1	186	134
4	59	52
6	40	34

Four concurrent streams, with 10% headroom, is this machine's capacity. Long context has little effect on generation speed: after 4K context, the median time per output token is 7.6 ms; after 256K it's 8.3 ms. What's really expensive is the first read of a long prompt—128K tokens takes 18.33 seconds—so prefix caching is critical. With a 256K prefix already cached and 4 users, 95% of requests get their first token within 1.66 seconds.

The seat model is also worth a look: one developer using Claude Code for a week had the model actually generate for about 2.8 hours, 1.7% of the week. Assuming each person independently has a 7.1% probability of being active within an 8-hour window, the probability that a 19-person team exceeds 4 concurrent streams is no more than 1%. But colleagues' schedules are not independent, so this estimate is optimistic, and it's based on only one person's usage data.

Prefix caching means prefilling only once the prompt prefix shared by multiple requests, and reusing the corresponding KV cache for subsequent requests. In agent scenarios, system prompts, tool definitions, and conversation history are often repeatedly concatenated into each request, and the prefix overlaps heavily, so the payoff from prefix caching is enormous. The article's 18.33 seconds for a 128K-token first read is exactly the cost of the prefill stage; once the prefix is cached, subsequent requests can begin decoding directly from the KV cache, and 95% of requests get their first token within 1.66 seconds. This is also why in long-context systems, cache hit rate often determines user experience more than the absolute speed of a single prefill.

Median and p5 are two percentile metrics for evaluating service quality. The median represents typical experience, while p5 (the 5th percentile) represents the speed of the slowest 5% of requests, reflecting tail-end experience. For agent sessions, a single extremely slow response can drag down an entire turn, so p5 is more informative than the average. In the table, 4 concurrent streams have a median of 59 tok/s and p5 of 52 tok/s, showing that experience is stable most of the time, but when assessing capacity you should use p5 rather than the median, and leave headroom.

After quantization, are the answers still correct

3 bit changes the model outputs, so it must be compared against a reference: fixed subsets of three benchmarks, thinking disabled, against the DeepSeek official API. GSM8K (math) is 0.960 on both sides. IFEval (instruction following) +0.013, interval [-0.013, +0.047]. HumanEval (code) -0.018, interval [-0.049, +0.012]. Both intervals include zero, and on around 150 questions, a one- or two-point difference is random fluctuation. The LiveBench agentic coding score of 77.3 (first among 69 models) is the unquantized model's result at maximum effort, not the quantized one's.

Problems not yet solved

All experiments involved only a single startup run, and the agent turns were synthetic rather than collected from real sessions. There is also no baseline comparison with "the entire model placed in a big machine's VRAM"—that experiment has been written up as E9. Method details are in the paper (PDF).

For those planning to equip a team with a coding agent machine, the significance of this scheme is not the headline "763B on two cards," but that it provides a reusable decision path: first look at the MoE's expert activation ratio, then do the math on host memory and PCIe bandwidth, and finally assess capacity per stream rather than for the whole machine.

One-line Summary

763B MoE on two RTX PRO 6000s: per-expert offload puts experts in host memory, and post-quantization accuracy aligns with the official API.

Paragraph Summary

The article recaps the complete process of deploying a 763B MoE model (DeepSeek-V4.1-Flash) on two RTX PRO 6000 Blackwells. After 3-bit quantization, each card still needs 102.1 GiB, exceeding the 95.6 GiB VRAM limit and causing OOM. The scheme exploits the fact that MoE activates only 6/384 experts per token, offloading 4,608 idle experts and lookup tables to host memory, read over PCIe via UVA, bringing per-card usage down to 72.4 GiB. vLLM's native UVA offload seems equivalent but crashes; what works is a combined stack of EXL3 + ExLlamaV3 + vllm-exl3 + Diffbot's per-expert plan. The article also emphasizes that PCIe/NUMA topology causes a 1.17 to 1.51x loss, that capacity should be measured per stream rather than by whole-machine throughput, and gives validation results showing that post-quantization differences on three benchmarks versus the official API all fall within confidence intervals.