One Machine Becomes Two: LLM Inference Disaggregation and the Engineering Logic Behind a 75% Throughput Gain

When input token volume in agent-style applications reaches 14 to 16 times the output token volume, the performance bottleneck in LLM inference is structurally clear: prefill and decode, two fundamentally different tasks, are crammed onto the same GPU and squeeze each other. In September 2026, two papers pushed this split deeper into model checkpoints and network architectures; and Prime Inference, launched by Prime Intellect on October 2, used production data from GB200 NVL72 showing "75% more requests on the same number of GPUs" to turn PD disaggregation from a paper concept into standard infrastructure.

Two Compute Modes, Two Hardware Requirements

LLM inference can be split into two phases with almost completely opposite characteristics. Understanding this asymmetry is the prerequisite for understanding the entire technical direction.

Prefill is compute-intensive. It forward-propagates the entire prompt in one pass and builds the KV cache, with attention computation growing quadratically with prompt length. It determines time to first token (TTFT)—how long the user waits after pressing Enter before seeing the first character.

Decode is memory-bandwidth-intensive. It generates only one token at a time, and each step must reload model weights and the growing KV cache from HBM. It determines inter-token latency (ITL)—the rhythm of streaming output.

Put both on the same GPU, and a long-prompt prefill freezes every in-flight decode. Agent workloads routinely have prompts above 100K tokens, and this blocking shows up most clearly in tail latency: most tokens look normal, then one token suddenly stalls for nearly a second. Chunked prefill (interleaving prefill slices with decode) mitigates this, while disaggregation eliminates it at the root—the decode pool never sees prefill.

KV cache is the material basis for understanding PD disaggregation. During autoregressive generation, the attention for each new token must look back at the Key and Value vectors of all previous tokens; without caching, every generated token would require recomputing the entire context, with cost growing linearly with sequence length. KV cache keeps the K/V tensors of every layer and every attention head resident in HBM, so the decode phase only appends and never recomputes. This has two direct consequences: first, cache size grows linearly with context length, layer count, head count, and precision, and a 100K-token agentic prompt can easily fill tens of GB of VRAM; second, each decode step must move all weights and the entire cache from HBM into the compute units, giving extremely low arithmetic intensity and making it bandwidth-bound rather than compute-bound. The "handoff" in PD disaggregation is essentially moving this KV cache produced by prefill from one machine to another, and its size and transfer time directly constitute a fixed additional cost in TTFT. Compressing KV precision (such as NVFP4) both increases the number of concurrent sessions a single GPU can hold and proportionally reduces transfer volume, making it one of the few techniques that optimizes both bottlenecks at once.

Chunked prefill is the mainstream mitigation before disaggregation, and its boundaries deserve separate explanation. It cuts the prefill of a long prompt into fixed-size chunks (for example, 2048 or 4096 tokens per step), concatenates them with in-flight decode steps into the same batch sent to the GPU, so that every scheduling cycle contains both prefill chunks and decode tokens, preventing one very long prefill from monopolizing SMs for an extended period. The cost is that prefill is stretched and TTFT rises, and chunk size becomes a tuning trade-off: larger chunks mean longer windows in which decode is blocked; smaller chunks mean heavier scheduling overhead and repeated weight loading. The more fundamental limitation is that chunked prefill only spreads out interference—prefill and decode still share the same HBM bandwidth and the same SMs, and under agent workloads with 14 to 16 times input bias, contention for resources cannot be eliminated. This is exactly why Prime Inference's reduction of the prefill budget from 8K to 4K tokens/step significantly lowers queue wait—it is still optimizing parameters within the same mechanism, whereas PD disaggregation changes the mechanism itself.

The Split Experiment: A Fivefold Gap in a Toy Simulation

The original author used a queue simulation to demonstrate the difference at the mechanism level (1B-class model, agentic-length prompts from 16K to 64K tokens, seed 42, explicitly labeled as a toy-level simulation).

At a load of 1 request per second, disaggregation reduced p99 inter-token latency from 5.0ms to 1.0ms (a 5x improvement), worst-case stalls from 931ms to 1ms, and p90 time to first token from 622ms to 448ms. When load rose to 2 requests per second, p99 ITL in the colocated system spiked to 263ms, while the disaggregated system remained stable at 1ms. KV transfer overhead accounted for about 3% to 4% of TTFT—this "toll" is real but small, and as prompts get longer, its relative share declines further.

This is also the most valuable judgment in the entire article: the handoff cost is the product itself. The maturity of KV transfer engineering directly determines whether disaggregation pays off.

Five Systems, One Continuously Deepening Split Line

A survey reviewed five representative systems along the disaggregation spectrum, all pointing in a highly consistent direction: from splitting machines to splitting the internals of models.

Splitwise (ISCA 2024, Microsoft Research) is the baseline approach: different machines, same model, achieving 1.4x throughput and 20% lower cost, and up to 2.35x under the same power and cost budget. Its core observation still holds: token generation does not need the compute capability of the latest GPUs, and there is no need to pay for compute you cannot use.

HMA-Serve (June 2026) moves toward splitting vendors: Tenstorrent Blackhole p150b cards (664 TFLOPS BFP8, about $1,300) run prefill, NVIDIA HBM GPUs run decode, combined with staged quantization (BFP8 for prefill, BF16 for decode) and compute-transfer pipeline overlap. It achieves 3.2x goodput and 4.8x goodput per dollar on Qwen3 4B to 32B with no quality loss. The economics are straightforward: an A100 wastes more than 97% of its HBM bandwidth during 4K-token prefill, and HBM accounts for 18% of A100 manufacturing cost and as much as 45% on B200—expensive bandwidth is only useful during decode.

SPAD (Princeton/University of Washington) splits the chip and designs from scratch: cutting HBM bandwidth by 40% makes prefill only 17% slower; cutting compute by 50% makes decode only 22% slower—each phase can tolerate losing the resources it does not use. Cluster simulations show hardware cost reductions of 19% to 41%. The original article explicitly notes the limitation: these are LLMCompass simulation results, not tape-out silicon, and should be treated as upper bounds rather than a price sheet.

Disaggregated Quantization (September 22, 2026) splits down to the checkpoint level, with a core finding of asymmetry: quantizing decode alone causes 2 to 4 times the accuracy loss of quantizing prefill alone (up to 7x on Gemma3-1B), because prefill needs fast low-precision math while decode only needs full weight loading. Its QADD training produces an NVFP4 prefill checkpoint paired with a separate decode checkpoint; on Qwen3.8-27B with 1-bit GGUF compression, MMLU-Pro improves by 32.5 points without touching the decode checkpoint. A third trick is streaming prefill weights from SSD in chunks, making TTFT 1.78x faster at 8K context.

KITE / Step Scale Transformer (September 23, 2026) splits into independent towers inside the architecture: the Prefiller tower builds the KV cache, the Decoder tower reads it, and newly added capacity never participates in KV construction, so scaling the model no longer scales prefill cost. The 67B MoE version reduces analytical reasoning cost by 31.6% versus a 63B classic Transformer while achieving better quality. Its motivation comes from the bill: in OpenRouter traffic from August 22 to September 20, the ratio of uncached input tokens to output tokens was 16.29x for Claude Fable 5 and 14.64x for GPT-6 Astra—input costs account for about 75% of total cost.

Goodput is the metric favored by this class of systems papers: the volume of valid requests or valid tokens per unit time that meet service objectives (such as TTFT and ITL SLOs), as distinct from raw throughput that does not distinguish whether timeouts occur. It embeds latency constraints into the denominator, so it can reflect the difference between "fast but useful" and "fast but queued." HMA-Serve's reported "goodput per dollar" further adds procurement cost as a dimension beyond the numerator: using low-bandwidth, low-cost accelerators on the prefill side, as long as their compute is sufficient to complete prefill within the SLO, raises effective output per dollar. Understanding this metric helps judge the comparability of various numbers—1.4x throughput, 3.2x goodput, and 75% more requests belong to different measures; the first measures raw capacity, while the latter two measure economic efficiency under a given latency target, and the latter is what production environments actually pay for.

Production Validation: Prime Inference's GB200 Tuning Data

Beyond research data, the production data Prime released on October 2 is more convincing. Its target workload is agentic: adding 6K tokens per round on top of a 140K-token prompt, benchmarked with SemiAnalysis AgentX. Key tuning conclusions include:

• A 1:4 prefill/decode ratio serves the most users: each prefill group carries 66 sessions, at 101 tokens/s per user
• DEP8 prefill topology provides about 5x the available prefix cache capacity of TEP8 on the same hardware
• Shrinking the prefill budget (from 8K to 4K tokens/step): median queue wait drops from 550ms to 110ms, and median TTFT falls by about 20%
• NVFP4 KV compression: each MLA row drops from 576 bytes to 352 bytes, and cached tokens per decoder rise from 1.09 million to 1.63 million
• Native sparse-MLA kernel: 12.0 microseconds at 15 query tokens, versus 17.7 microseconds for the staged approach and 13.7 microseconds for the FP8 approach
• BLHNC KV layout: transfer descriptors drop from 19,559 to 1,940, and average transfer time falls from 146ms to 78ms

Progress at the packaging layer is equally important: llm-d wraps PD disaggregation as Kubernetes objects, and GPU virtualization lets a single card share about 2x the number of decode instances, yielding about 20% higher cluster throughput. This split is also extending beyond prefill/decode—vision encoders are being separated from the LLM, and MoE systems are separating attention from FFN.

The DEP8 and TEP8 that appear repeatedly in the article are names for two different parallel topologies on GB200 NVL72. The letter combinations usually indicate the arrangement of parallel dimensions: E stands for expert parallel (expert sharding for MoE models), P stands for pipeline parallel (splitting layers across devices), and D and T correspond to data parallel and tensor parallel respectively. DEP8 means organizing prefill or decode serving across 8 GPU instances within the same parallel group using a combination of data/expert/pipeline parallelism. Different topologies determine how KV cache and prefix cache are distributed across devices: if the prefill topology spreads requests across more independent replicas, each replica can cache a more complete prefix, which explains why DEP8's available prefix cache capacity is about 5x that of TEP8. Prefix cache hit rate translates directly into saved prefill computation, so in agent multi-turn conversations with high reuse, topology choice itself is a cost decision, not just a deployment detail.

A Few Takeaways You Can Use Directly

Running one request per GPU is a compromise for both phases. PD disaggregation has become standard infrastructure: NVIDIA documents it as a regular Dynamo configuration, and cloud providers list it as a baseline option.

The split keeps going deeper. From machines (2024) to vendors (mid-2026) to checkpoints and architectures (September 2026). A sharp question follows: when the two phases run on different silicon, different numeric formats, and different network towers, what does "serving a model" even mean?

Prefill is the bill. Agent traffic shows a 14 to 16 times input bias; every prefill optimization—budget control, prefix caching, NVFP4 KV, SST-style architectures—lands directly on cost.

Don't bother at small scale. A well-tuned single vLLM pool is simpler and perfectly adequate under light traffic. Disaggregation pays off only when serving a single model at large scale and users can perceive the streaming SLO.

One-line Summary

PD disaggregation splits prefill and decode onto different hardware, trading a 3% to 4% KV transfer cost for structural improvements in latency and throughput.

Paragraph Summary

The article argues that prefill (compute-intensive, determines TTFT) and decode (bandwidth-intensive, determines ITL) sharing a GPU blocks each other, especially under agent workloads where input is 14 to 16 times output. A queue simulation shows disaggregation reducing p99 ITL from 5.0ms to 1.0ms and worst-case stalls from 931ms to 1ms, while KV transfer accounts for only 3% to 4% of TTFT. It then reviews the disaggregation spectrum across five systems: Splitwise splits machines, HMA-Serve splits vendors, SPAD splits chips, Disaggregated Quantization splits checkpoints, and KITE splits the internals of the architecture. Finally, Prime Inference's production tuning on GB200 NVL72 (1:4 ratio, DEP8 topology, NVFP4 KV, sparse-MLA kernel) supports 75% more requests on the same number of GPUs.