Multi-Turn Memory, Mixed Precision, and Safer Rollouts: How Serving Is Catching Up to Agentic Workloads

Agentic workloads are rewriting the underlying assumptions of inference serving. For years, the entire industry optimized for the "single request" scenario, but once LLMs become agents that hold ongoing conversations and repeatedly call tools, the bottleneck shifts from compute-bound to memory- and scheduling-bound. This article surveys a batch of recent developments emerging across arXiv, vLLM, Hugging Face, Modal, and elsewhere—all pointing in the same direction: evolving inference serving from "finish one request" to "make a whole session cheap to run." Figure: Multi-Turn Memory, Mixed Precision, and Safer Rollouts — Serving Catches Up to Agentic Workloads

Multi-Turn Sessions: Bidirectional Flow of KV Cache

The most direct pain point of multi-turn agentic workloads is that TTFT (time to first token) gets dragged down by repeatedly recomputed prefill. Traditional KV cache only flows from prefill nodes to decode nodes, so every new turn of a conversation requires re-prefilling the entire history. vLLM's latest work breaks this one-way channel—letting decode nodes "return" already-computed KV blocks to prefill nodes, so the shared prefix of the next conversation turn no longer needs to be recomputed.

According to the vLLM blog, this bidirectional KV transfer cuts TTFT for a 24k token scenario on Qwen3-32B from 778ms to 460ms (about 1.6x), and for a 20k token scenario on GLM-5-FP8 from 2334ms to 702ms (about 3.3x). The key is that it requires no additional hardware—it works with a KV-cache-aware router and EFA-based point-to-point tensor transfer. For any team deploying long-session agents, this is a direct latency and cost win.

KV Cache and the Two Phases of Prefill/Decode: During Transformer inference, the attention computation for each token depends on the Key (K) and Value (V) vectors of all preceding tokens; these intermediate results are the KV cache. Inference is split into two phases: prefill processes the entire input prompt in parallel at once, generates the first token, and fills the KV cache—it is compute-bound; decode generates tokens autoregressively one by one, reading the entire KV cache at each step—it is memory-bandwidth-bound. In multi-turn conversations, the KV already computed for prior dialogue traditionally resides only on decode nodes, yet the newly added prefix of the next turn still has to be recomputed on prefill nodes. TTFT (Time To First Token) is mainly determined by prefill duration, making it the most sensitive user-experience metric for long sessions. Passing computed KV blocks back from decode to prefill for reuse is essentially eliminating this redundant computation.

EFA Point-to-Point Transfer: EFA (Elastic Fabric Adapter) is AWS's network interface for high-performance computing and distributed training, supporting kernel-bypass RDMA-style point-to-point communication and GPUDirect, allowing KV blocks to move between nodes at near-memory-bandwidth speeds without going through CPU or shared storage. This is precisely the engineering premise that lets bidirectional KV flow avoid extra specialized hardware—reusing the cluster's existing high-speed interconnect to bring cross-node KV migration costs into an acceptable range.

Scheduling Layer: The Trade-off Between Session Affinity and Queue Congestion

If vLLM solves the problem from inside the serving engine, Vesta on arXiv approaches it from scheduling theory. It points out a neglected structural problem: decoupling "session-level KV cache ownership" from "request-level execution placement"—that is, not every request must be routed to the node holding its cache; instead, queueing theory is used to trade off cache locality against queue congestion.

The paper reports a 14% throughput improvement and 57.8% reduction in average TTFT under co-located deployment, and up to 29% reduction in P99 TTFT under disaggregated deployment. This result complements vLLM's bidirectional KV transfer: one reduces redundant computation from the engine side, the other reduces queueing wait from the scheduling side. Both roads lead to the same destination—a "session-first" serving architecture.

Session Affinity and Queueing Theory: Session affinity means routing subsequent requests of the same session to the same node holding its KV cache, so the cache is hit and recomputation is avoided. But strict binding has side effects: if the target node is busy, requests can only wait in queue, worsening tail latency; while loose routing can spread load but may miss the cache every time. Vesta turns this into an optimizable problem using queueing theory—decoupling "where the cache lives" (session ownership) from "where the request executes" (execution placement), allowing requests to be sent to idle nodes for execution while maintaining hit rates through cache migration or prefetching, striking a balance between cache locality and queue congestion. The value of this perspective is that it turns scheduling from heuristic rules into a quantifiable cost trade-off, and explains why co-located deployment (cache and compute on the same node) and disaggregated deployment (prefill/decode separation) require different scheduling strategies.

Mixed Precision: No More Guessing Configurations by Byte Width

KV cache quantization strategy has always been the most hand-wavy part of inference optimization. The paper "The Price of Precision" proposes a calibrated cost model: first determine whether the real bottleneck under a given batch size and context length is computation, weight loading, or KV cache access, then use dynamic programming to select a mixed-precision configuration rather than relying solely on byte width. For teams that need to tune KV memory limits model by model, this is far more practical than a one-size-fits-all bit count.

In the same direction is MultiAlloc, which splits a fixed quantization budget into two parts—"per-layer precision boosting" and "per-token retention"—using lightweight offline profiling instead of uniform compression. Under equal memory budget, it outperforms single-granularity baselines on Mistral, Llama, and QwQ. E8 Lattice Key Quantization takes a more aggressive route: replacing scalar quantization with vector quantization on the E8 lattice, exploiting cross-dimensional correlations to achieve a 2-bit keys / 2-bit values configuration (previously keys required at least 3-bit), with about 6x compression while decode throughput matches the scalar baseline.

Granularity and Budget of KV Quantization: Compressing KV cache from FP16 to lower bits significantly reduces memory footprint and increases batch size, but introduces precision loss. Quantization granularity determines compression efficiency and error distribution: scalar quantization rounds each value independently, simple to implement but ignoring inter-dimensional correlations; vector quantization maps several dimensions as a whole to a codebook, exploiting correlations for higher compression ratios. The E8 lattice is one of the optimal sphere-packing structures in eight-dimensional space; using it for vector quantization maintains low reconstruction error at extremely low bit widths—this is the mathematical basis enabling 2-bit keys. The core question of mixed-precision strategy is "how to allocate the budget"—whether to uniformly compress all layers and all tokens, or retain higher precision for sensitive layers and important tokens. MultiAlloc's layered budget and per-token retention, along with "The Price of Precision"'s cost model, are both answering this question.

Rollout Safety: Mirror Traffic First, Then Switch Production

Once a faster serving path is chosen, teams face another problem: how to validate it safely. Modal's Endpoint Candidates feature offers a built-in solution: let a standby deployment (a new checkpoint or a different model) mirror a sampled portion of production traffic, measuring quality, latency, throughput, and cost before promoting it to a formal endpoint.

Its engineering implementation is worth noting—Modal extended its Pingora-based proxy so that each mirrored candidate reads from its own bounded, deduplicated buffer, with process-level memory budgets, ensuring slow candidates don't add latency to production traffic. For teams wanting to A/B test quantization adjustments, model upgrades, or engine configuration changes without building their own traffic-mirroring infrastructure, this is a low-risk path.

Hardware and Sparsification: The Next Stage of Agentic Throughput

Evidence shows that vLLM's three optimizations for DeepSeek^Ad-V4.1-Flash—SWA bounded replay (recomputing only the last 128 tokens), MegaAttention and Mega-Gate kernels, and operator fusion with CUDA graphs—achieve 5.3x throughput under a 150 TPS constraint, 1.9x speedup at low concurrency, and about 70% lower TTFT at 100K throughput. This means a step-change in cost per token.

On the hardware side, vLLM's day-0 support for NVIDIA Vera Rubin NVL72 reports 7.8x throughput per GPU (AgentX) relative to GB200 NVL72 and 3.7x VLM throughput (MLPerf v6.1) relative to GB300, with "locality domains" tuned for MoE routing. This is more a preview of future hardware than a solution deployable today, but it has reference value for hardware planning.

On the model side, LiquidAI's LFM2.5-24B-A2B uses sparse MoE to activate only 2B of its 24B parameters per token, and expands context from 32K to 131K. It achieves 184 tok/s on an Apple M5 Max and up to 21K tok/s aggregate throughput on a single NVIDIA B300, providing a direct data point for "running low-cost agents on a single GPU or on-device."

Sparse MoE and Activated Parameter Count: Mixture-of-Experts (MoE) models split the feed-forward layers into multiple experts, with each token routed to only a few of them. Thus "total parameter count" and "activated parameters per token" are two different concepts. The naming LFM2.5-24B-A2B illustrates this: 24B total parameters, only 2B activated per token, so inference compute is close to a 2B dense model while enjoying the capacity of larger parameters. This allows it to achieve high throughput even on a single B300 or Apple M5 Max, providing a data point for low-cost agent deployment on-device or on a single GPU. The cost of MoE is that memory must still hold all expert weights, and routing itself introduces extra communication and scheduling overhead—which is why NVIDIA emphasizes "locality domains" on Vera Rubin NVL72: keeping routing within local high-speed interconnect range to avoid expert cross-node transfers becoming a bottleneck.

The Throughline of the Week

The common logic of this wave of progress is that serving systems are being redesigned for "sessions" rather than "requests." vLLM's bidirectional KV transfer and the Vesta scheduler attack the same problem from the engine and scheduling sides—multi-turn sessions repeatedly paying prefill costs, or getting stuck in naive queueing. Meanwhile, Modal's Endpoint Candidates and the DeepSeek-V4.1-Flash kernel work point to the latter half: once you've chosen a faster serving path, you need a safe way to validate it on real traffic before fully switching over.

For inference infrastructure teams, this means the evaluation framework should also be updated: no longer looking only at single-request TTFT and throughput, but at session-level cost, cache hit rate, and P99 tail latency.

One-line Summary

Inference serving is shifting from optimizing single requests to optimizing entire sessions, with engine, scheduling, quantization, and rollout safety advancing in parallel.

Paragraph Summary

This article surveys a batch of recent developments in agentic inference serving, with the core logic that serving systems are being redesigned for "sessions" rather than "requests." vLLM introduces bidirectional KV transfer, letting decode nodes return KV blocks to prefill nodes, cutting TTFT by up to 3.3x in multi-turn scenarios without extra hardware; arXiv's Vesta scheduler decouples session KV ownership from request execution placement, trading off cache locality against queue congestion, reducing average TTFT by 57.8%. On the quantization side, "The Price of Precision" proposes a calibrated cost model with dynamic programming to select mixed precision, MultiAlloc does layered budgeting, and E8 Lattice uses quantization to achieve 2-bit KV. On the engineering side, Modal Endpoint Candidates provides built-in traffic-mirroring canary. On the hardware and model side, DeepSeek-V4.1-Flash achieves 5.3x throughput on vLLM, and the sparse MoE model LFM2.5-24B-A2B reaches 21K tok/s aggregate throughput on a single GPU.