vLLM Introduces Push-Based KV Transfer: 3x TTFT Improvement in Disaggregated Inference

vLLM's prefill-decode disaggregated architecture splits LLM inference into two independent stages, but KV cache transfer between stages often becomes a performance bottleneck. Recently, the vLLM team introduced a push-based NIXL connector that moves transfer coordination off the critical path. In benchmarks on Amazon EC2 P5en instances with AWS EFA, this approach achieved up to 3x improvement in TTFT (time to first token) and a 30% improvement in TPOT (time per output token).

The Bottleneck of Disaggregated Inference: KV Cache Transfer

In vLLM's disaggregated inference, the prefill stage processes user input and generates the KV cache, while the decode stage generates output token by token based on the cache. The two stages run on different engine instances, so the KV cache must be transferred over the network. In the traditional pull-based mode, the decode instance actively initiates transfer requests to the prefill instance, placing coordination overhead on the decode-side critical path and directly slowing down first-token generation.

The KV cache consists of the key and value tensors that Transformer models store during autoregressive generation to avoid recomputing already-processed tokens. In the prefill stage, the model computes and stores KV caches for all layers of the entire input sequence; the decode stage reuses these caches and only computes attention for new tokens. As a result, KV cache size is proportional to sequence length, number of layers, and hidden dimension, reaching tens of GB for long-context models. In disaggregated inference, this data must be transferred over the network from the prefill instance to the decode instance, and its transfer latency directly affects time to first token (TTFT).

Push Mode: Moving Coordination Off the Critical Path

The core of the new approach is changing the direction in which data transfers are initiated. After completing KV cache generation, the prefill instance actively pushes the cache to the decode instance, and the decode instance simply waits for the data to arrive. This way, transfer coordination (such as connection establishment and metadata exchange) no longer blocks the decode instance's token generation loop. The NIXL connector, as the underlying transport library, is responsible for efficiently managing memory and network resources.

According to the vLLM team, push mode reduces idle waiting time and is especially suitable for high-concurrency scenarios. The decode instance can begin processing the next request earlier, thereby improving overall throughput.

NIXL (NVIDIA Inference Xfer Library) is a high-performance data transfer library designed specifically for AI inference scenarios, supporting GPUDirect, RDMA, and multiple network backends. It provides low-latency, high-throughput asynchronous transfer primitives and allows applications to customize transfer coordination logic. In vLLM's push-based connector, NIXL manages memory registration, connection establishment, and data movement, enabling the prefill instance to efficiently push KV caches to the decode instance without frequent polling or waiting on the decode side.

Benchmarks: Improvements in Both TTFT and TPOT

On Amazon EC2 P5en instances using AWS EFA (Elastic Fabric Adapter) for inter-node communication, the team compared the performance of push-based and pull-based connectors. Results showed up to 3x improvement in TTFT and about 30% improvement in TPOT. This means users see the first output token sooner, and subsequent tokens are generated at a tighter pace.

These improvements are particularly important for latency-sensitive applications such as chatbots and real-time translation. Lower TTFT directly improves the interactive experience, while TPOT optimization makes long-text generation smoother.

AWS EFA (Elastic Fabric Adapter) is a network interface provided by Amazon Web Services, designed for high-performance computing (HPC) and machine learning workloads. It supports OS bypass and RDMA (remote direct memory access), significantly reducing inter-node communication latency and improving bandwidth utilization. On EC2 P5en instances, EFA provides the underlying network support necessary for multi-node distributed inference, ensuring that large-scale KV cache transfers do not become a bottleneck. Combined with the NIXL library, EFA can fully leverage its low-latency characteristics, thereby amplifying the advantages of push mode.

Impact on the vLLM Ecosystem

Push-based KV transfer is an important optimization by vLLM in the direction of disaggregated inference. It does not change model structure or inference logic; instead, it unlocks hardware potential by adjusting the transfer coordination mechanism. For teams already deploying disaggregated inference, upgrading to the new connector may bring immediate latency benefits.

However, push mode places higher demands on the prefill instance's active push capability, requiring proper network bandwidth and buffer management. The vLLM team provides configuration guidance in their blog and recommends that users adjust parameters according to actual workloads.

As disaggregated inference becomes increasingly popular in low-cost, high-throughput scenarios, KV transfer efficiency will become a key metric distinguishing inference framework performance. This optimization by vLLM offers the community a reusable idea: removing coordination logic from the critical path is often more effective than simply adding bandwidth.

One-line Summary

vLLM introduces a push-based NIXL connector that moves KV cache transfer coordination off the decode critical path, improving TTFT by 3x.

Paragraph Summary

In vLLM's prefill-decode disaggregated architecture, KV cache transfer often becomes a performance bottleneck. In the traditional pull mode, the decode instance actively requests transfers, and coordination overhead blocks token generation. The new approach uses push mode, where the prefill instance actively pushes the cache and the decode instance simply waits for data to arrive, moving coordination off the critical path. The underlying NIXL connector manages memory and network resources. In benchmarks on Amazon EC2 P5en with AWS EFA, the push-based connector achieved up to 3x TTFT improvement and about 30% TPOT improvement, especially suitable for high-concurrency, latency-sensitive scenarios. This optimization does not change model structure and provides a reusable performance improvement approach for disaggregated inference.