vLLM Bidirectional KV Transfer Optimizes Multi-Turn Agent TTFT

Multi-turn agent workloads need to reprocess the growing conversation context each round, keeping time to first token (TTFT) persistently high. The bidirectional KV cache transfer mechanism introduced by vLLM lets prefill and decode nodes reuse already-computed context across rounds. On Amazon EC2 P5en instances with EFA networking, TTFT improves by up to 1.6x for Qwen3-32B and reaches 3.3x for GLM-5-FP8.

Where the TTFT Bottleneck in Multi-Turn Agents Comes From

A typical agentic workload looks like this: the user initiates a task, the model calls a tool, the tool returns a result, and the model continues reasoning. Each round feeds the historical context back into the model, and the prefill-stage computation grows linearly with the number of rounds. In traditional PD-disaggregated architectures, prefill nodes and decode nodes operate independently, and the KV cache cannot be passed across rounds—so whatever was computed last round has to be computed again this round.

This redundant computation has limited impact in short conversations, but it accumulates quickly in multi-turn agent scenarios. The longer the context, the more noticeably TTFT degrades, eventually becoming a hard constraint on interaction experience. The root cause is not insufficient compute power but the lack of a cache reuse mechanism.

TTFT (Time To First Token) refers to the time from when a request is sent to when the model outputs the first token, and is the core metric for measuring response speed in interactive inference. In agent scenarios, it directly determines the startup latency of the tool-calling loop. In contrast, TPOT (Time Per Output Token) measures the average generation time per token during the decode phase. In multi-turn conversations, because historical context must be re-prefilled, TTFT degrades significantly as the number of rounds and context length grow, while TPOT remains relatively stable. Therefore, the key to optimizing multi-turn agent experience lies in controlling TTFT rather than simply improving decode throughput.

The difference in computational characteristics between prefill and decode is the premise for understanding PD disaggregation. The prefill phase performs a single forward computation over the entire input, producing KV caches for all positions; it is compute-bound and highly parallelizable. The decode phase generates tokens autoregressively one at a time, processing only one token per step but needing to read the entire historical KV cache; it is memory-bandwidth-bound. The two have completely different hardware resource requirements and interfere with each other when co-located—which is precisely the starting point for PD-disaggregated architectures.

How Bidirectional KV Transfer Works

vLLM's approach is to establish a bidirectional KV cache channel between prefill nodes and decode nodes. When a round of generation ends, the KV cache held on the decode node is sent back to the prefill node and reused directly at the start of the next round, skipping recomputation of the existing context.

The "bidirectional" part means the transfer is not one-way: prefill passes the initial KV to decode, and the incremental KV produced by decode during generation also flows back to the prefill side. This way, each round performs incremental prefill on top of the previous round rather than a full recomputation. Combined with the high-bandwidth, low-latency EFA (Elastic Fabric Adapter) network, inter-node transfer overhead is kept within an acceptable range.

Synergy with PD-Disaggregated Architecture

PD disaggregation (Prefill-Decode Disaggregation) splits prefill and decoding onto different nodes, originally to let each type of computation be optimized separately. But after disaggregation, the KV cache is fragmented, which actually amplifies redundant computation in multi-turn scenarios. Bidirectional KV transfer fills this gap, allowing PD-disaggregated architectures to maintain efficiency under multi-turn agent workloads as well.

KV cache (Key-Value Cache) is a key data structure in Transformer inference. The self-attention mechanism needs to compute attention against the Key and Value vectors of all historical tokens when generating each token. To avoid redundant computation, inference engines cache these K/V vectors. The size of the KV cache is proportional to the number of layers, number of attention heads, head dimension, sequence length, and batch size, and can consume tens of GB of GPU memory in long-context scenarios. In multi-turn agents, whether the KV cache can be reused across rounds directly determines whether repeated prefill is necessary. The essence of bidirectional transfer is to make this cache a state that can be migrated between nodes, rather than temporary data bound to the lifecycle of a single node.

EFA (Elastic Fabric Adapter) is a high-performance network interface provided by AWS for EC2 instances, designed specifically for HPC and distributed AI training/inference. Through an OS-bypass mechanism, it lets applications access network hardware directly, bypassing the kernel protocol stack, thereby significantly reducing latency and improving bandwidth utilization, while supporting the SRD (Scalable Reliable Datagram) protocol for multipath transmission. In bidirectional KV transfer scenarios, the KV cache needs to travel back and forth frequently between prefill and decode nodes; the data volume is large and latency-sensitive, and EFA's low-latency, high-bandwidth characteristics keep this overhead from offsetting the gains of cache reuse. On ordinary TCP networks, cross-node KV transfer latency could become a new bottleneck.

Measured Data: Clear Differences in Gains Across Models

Official tests on Amazon EC2 P5en instances show that gains vary by model:

• Qwen3-32B: TTFT improves by up to 1.6x
• GLM-5-FP8: TTFT improves by up to 3.3x

FP8-quantized GLM-5 shows significantly higher gains. A reasonable explanation is that the FP8 model has higher per-prefill computational throughput, so the absolute time share of repeated prefill is larger, making the gains from eliminating recomputation more pronounced. Model size, quantization precision, and context length all affect the final multiplier; 3.3x is a peak under a specific configuration, not a universal value.

Implications for Agent Application Development

The value of this optimization is not limited to the inference side. For teams building agent applications, TTFT directly determines the response speed of the tool-calling loop. In multi-turn tasks, each round has to wait for the model to "recall" history, repeatedly interrupting the user experience. Only by driving TTFT down can the interaction rhythm of agents approach that of real-time conversation.

Another point worth noting is architecture choice. If a team is already deploying vLLM with PD disaggregation, bidirectional KV transfer is a natural upgrade; if still deploying on a single node, the network dependency introduced by cross-node transfer needs to be considered—high-speed interconnects like EFA are not available in every environment. The trade-off between gains and deployment complexity depends on the depth of the agent workload's rounds and the scale of the context.

From a broader perspective, the KV cache is evolving from "temporary state during inference" into a first-class resource that can flow across rounds and across nodes. Reuse, transfer, and eviction strategies around the KV cache will become a competitive focus of agent inference infrastructure. vLLM's bidirectional transfer this time is a clear signal along this path.

One-line Summary

vLLM's bidirectional KV cache transfer lets multi-turn agents reuse context across rounds, improving TTFT by up to 3.3x.

Paragraph Summary

To address persistently high TTFT in multi-turn agent workloads, vLLM introduces a bidirectional KV cache transfer mechanism between prefill and decode nodes. In traditional PD-disaggregated architectures, the KV cache is fragmented, requiring full recomputation of historical context each round and causing prefill computation to grow linearly with rounds. Bidirectional transfer returns the incremental KV produced by decode nodes to the prefill side, enabling incremental prefill across rounds rather than full recomputation. In tests on Amazon EC2 P5en instances with high-speed EFA networking, TTFT improves by up to 1.6x for Qwen3-32B and 3.3x for FP8-quantized GLM-5. The solution fills the cache-fragmentation gap of PD disaggregation in multi-turn scenarios, but also introduces a dependency on high-speed interconnects. The KV cache is becoming a first-class resource that can flow across rounds and nodes.