TB/s Memory Tier Accelerates SGLang: How X-Mem Reduces TTFT by 6.7x

When KV Cache Bandwidth Becomes the New LLM Inference Bottleneck

In large model inference, GPU compute has long been considered the scarcest resource. Yet as long-context and multi-turn conversation use cases go mainstream, a long-overlooked bottleneck is coming into focus — the memory bandwidth of KV caches. A recent technical blog from Netpreme sends a clear signal: in high cache hit rate scenarios, the ceiling on inference performance is no longer compute, but the speed at which data can be moved through memory.

The core of this blog is integrating their proprietary X-Mem™ MPU (Memory Processing Unit) as a TB/s-class memory tier into SGLang's HiCache layered KV cache system. The result is a substantial performance improvement — and a fresh perspective on the memory architecture of inference systems.

KV Cache: From Compute Optimization to Bandwidth Bottleneck

To understand why memory bandwidth becomes a bottleneck, we first need to clarify what KV cache actually is. KV cache (Key-Value Cache) is a core optimization technique for Transformer inference. During autoregressive generation, each new token requires attention computation against all previous tokens. Without caching, the Key and Value matrices of historical tokens must be recomputed at every step — an overhead that grows quadratically with sequence length. KV cache eliminates this by storing previously computed K/V matrices for reuse, reducing computational complexity from O(n²) to O(n).

The trade-off, however, is that GPU memory consumption grows linearly with context length. Take Llama-3 70B as an example: processing a 32K context can require tens of gigabytes of KV cache alone — far exceeding the memory footprint of the model weights themselves. This is the fundamental driver behind the memory pressure surge in long-context scenarios, and the direct motivation for introducing tiered caching strategies in inference systems.

The Real Bottleneck in High Hit Rate Scenarios

So why does memory bandwidth suddenly matter so much? The answer lies in specific application patterns.

The blog uses multi-turn coding sessions with Claude Code on SWE-bench as its reference scenario. These tasks are characterized by developers repeatedly querying, iterating, and modifying the same codebase across turns — resulting in highly overlapping context between turns. This overlap directly drives extremely high prefix cache hit rates, averaging around 98%.

It's worth understanding how prefix caching works. Modern inference frameworks like SGLang and vLLM implement hash-based prefix caching (e.g., SGLang's RadixAttention): the system computes hashes over chunks of the input token sequence, and on a cache hit, directly reuses existing KV tensors — completely skipping redundant prefill computation. In Claude Code-style scenarios, where users repeatedly query the same codebase, the vast majority of token prefixes are identical. A 98% hit rate isn't an anomaly — it's an intrinsic property of the application.

When hit rates approach saturation, the system barely needs to recompute KVs at all. It simply needs to move cached KV data from the storage tier back to the GPU as quickly as possible. This means compute units are mostly sitting idle waiting for data — the true bottleneck shifts from compute to the bandwidth of KV data movement. At this point, adding more GPU compute does almost nothing. What determines the user experience is how fast the memory tier can feed data to the model.

The Media Selection Problem and the Bandwidth Cliff

HiCache is SGLang's tiered KV cache system, inspired by the classic memory hierarchy. It stores KV data across tiers based on access frequency: the hottest data stays in GPU HBM (high-bandwidth memory), warm data is offloaded to Host DRAM, and cold data can be further persisted to NVMe SSDs.

The core challenge in this architecture is the "bandwidth cliff": GPU HBM bandwidth typically operates in the 2–3 TB/s range (e.g., 3.35 TB/s for the H100), while the effective bandwidth of Host DRAM connected over PCIe 5.0 is only around 64 GB/s — a nearly 50× gap. Traditional solutions use Host DRAM as the offload tier. When large volumes of KV data need to be reloaded from DRAM back to the GPU, this bandwidth gap manifests directly as significant TTFT degradation.

Netpreme's approach is to replace this DRAM tier with X-Mem™, inserting a higher-bandwidth memory medium at this exact position. Notably, this substitution is implemented via HiCache's existing tiering interface, making it a drop-in integration that requires no changes to upstream inference logic — an extremely low engineering overhead for deployment.

X-Mem™ MPU: A Dedicated Memory Unit Optimized for Data Movement

An MPU (Memory Processing Unit) is a class of specialized processing unit whose core design goals are memory bandwidth and capacity optimization — distinct from GPUs or TPUs, which prioritize compute throughput. In traditional servers, the CPU and GPU communicate over PCIe, with bandwidth constrained by the bus specification. MPUs, by contrast, are typically connected via high-speed interconnects like CXL (Compute Express Link) or deployed as near-memory compute units, capable of supplying data to the GPU at TB/s-class bandwidth — fundamentally bridging the bandwidth gap between GPU HBM and host memory.

X-Mem™ is Netpreme's proprietary memory processing product, and its core value proposition is delivering effective host-side bandwidth far exceeding conventional DRAM. The emergence of such products reflects a broader industry trend toward specialization of the memory subsystem in disaggregated compute architectures — when data movement itself becomes the performance bottleneck, hardware purpose-built for data transfer carries unique system-level value.

Key Performance Numbers Delivered by X-Mem™

According to data from Netpreme's blog, switching the offload tier from Host DRAM to X-Mem™ yields significant improvements across multiple core metrics:

• TTFT (Time to First Token) reduced by up to 6.7×: First-token response speed directly impacts user experience; a 6.7× improvement is nearly an order-of-magnitude gain.
• Single-user TPS improved by 33%–50%: The token generation throughput available to each user increases substantially.
• Overall system throughput improved by 30%: The same hardware can serve more concurrent requests.
• Drop-in integration: Seamlessly integrated via the HiCache tiering interface, with no changes required to inference logic.

The logic behind these numbers is consistent: when bandwidth is the bottleneck, increasing memory tier bandwidth translates directly into end-to-end performance gains — and is often more effective than pure compute scaling.

Why TTFT Shows the Most Dramatic Improvement

Understanding this contrast requires distinguishing the technical meaning of two key metrics. TTFT (Time To First Token) measures the time from when a user sends a request to when the first output token is received, and is primarily determined by the prefill phase. In high cache hit scenarios, the prefill phase is mostly occupied with loading cached KV data into the GPU — computationally trivial, but data-movement intensive, making it highly sensitive and nearly linearly responsive to memory bandwidth. Under DRAM bandwidth constraints, latency accumulates rapidly; with a TB/s-class memory tier, this bottleneck is directly eliminated, which is why the TTFT gain is most dramatic (6.7×).

TPS (Tokens Per Second), by contrast, measures steady-state generation throughput, reflecting the combined performance of the decode phase — including computation, KV reads, scheduling, and more. Eliminating a single bottleneck in a multi-stage system yields a more distributed improvement, so the gains are more moderate (30%–50%) but still substantial. The stark contrast between TTFT and TPS improvement magnitudes is precisely the evidence that validates the hypothesis: memory bandwidth is the primary bottleneck.

Implications for Inference System Architecture

The greatest value of this work may not lie in any specific number, but in the trend it reveals: LLM inference systems are evolving from compute-bound to memory-bound.

As agents, long-context workloads, and multi-turn conversations become mainstream, both the scale and reuse rate of KV caches are climbing rapidly. When cache hit rates approach saturation, system efficiency becomes entirely dependent on the design of the memory subsystem. This opens a much larger space for specialized memory hardware (such as MPUs) in inference infrastructure — they are no longer optional accessories, but critical components that determine end-to-end performance.

For teams building inference services, this case study offers two practical takeaways:

1. Profile for bottlenecks before optimizing: In high cache hit rate scenarios, blindly adding GPUs may yield minimal returns. Identifying the bandwidth bottleneck is the prerequisite for meaningful optimization. Monitor the ratio of KV load time during the prefill phase to GPU utilization to determine whether your system has entered the bandwidth-bound regime.
2. Pay attention to storage media selection in tiered caches: Systems like HiCache that expose a tiering interface allow storage media to be swapped without changing upstream logic, offering tremendous flexibility for incremental optimization. When evaluating options, the bandwidth ratio between GPU HBM and the offload tier is a system parameter worth close attention.

Conclusion

Netpreme's blog is a textbook example of "bottleneck migration" — a reminder that as LLM application patterns evolve, the battleground for performance optimization keeps shifting. In high cache reuse scenarios like multi-turn coding sessions, memory bandwidth has become the new critical variable. The combination of X-Mem™ and SGLang HiCache uses a solid set of performance numbers to demonstrate the value of a dedicated memory tier: not more compute, but faster data movement, is the true lever for performance gains in these workloads.

A note: the data above comes from the vendor's own technical blog, and real-world benefits should be independently validated in production environments. That said, the era of "memory as performance" is arriving faster than many expect.

Key Takeaways