Calibrate, Then Route: Intelligent Request Scheduling for Disaggregated LLM Serving

The Routing Challenge in Disaggregated LLM Serving

LLM inference serving is undergoing a fundamental architectural shift. To address the conflicting hardware demands of the compute-intensive prefill phase and the memory-intensive decode phase, the industry has introduced disaggregated deployment — placing these two phases on separate GPU resource pools. Systems like DistServe, Splitwise, and Mooncake have made this disaggregation fast enough, but one critical question remains unanswered: when a request arrives, which instance should handle it?

This is the core problem that routing strategies must solve. A newly published arXiv paper, Calibrate, Then Route, presents a systematic empirical study of this problem, proposing a learning-based router grounded in cost estimation and validating it on real hardware.

The prefill phase processes all tokens in the user's input through a large-scale parallel matrix computation — an arithmetic-intensive task sensitive to GPU compute throughput. The decode phase autoregressively generates output tokens one at a time; each step computes only a single token but must repeatedly read the KV cache (Key-Value Cache) of all previously generated tokens, making it a memory-bandwidth-intensive task. These fundamentally different hardware requirements cause mutual interference when both phases run on the same instance: prefill stalls decode, degrading both Time To First Token (TTFT) and Time Per Output Token (TPOT). Disaggregated deployment assigns each type of workload to a dedicated GPU pool, allowing hardware configurations and batching strategies to be independently optimized, thereby improving overall inference efficiency and service quality.

How the Router Makes Decisions

The core idea in the paper is to have the router estimate the "additional completion time" on each instance and use that estimate to make assignment decisions. This estimation draws on four dimensions of information:

• Exact prompt length: the number of input tokens in the request is a known, deterministic value
• Predicted output length: the number of tokens the decode phase will generate needs to be estimated
• Post-admission KV cache pressure: the impact on GPU memory cache once the request is admitted
• SLO class: the service-level objective category the request belongs to

This approach essentially upgrades routing from simple load balancing (e.g., round-robin, least-load) to explicit modeling of the true cost of each request. The researchers first developed the strategy in a discrete-event simulator, then transferred it to real hardware for validation — this "simulate first, validate second" methodology is itself one of the paper's contributions.

Performance on Real Hardware

The evaluation setup is rigorous: eight NVIDIA A40 GPUs, each running a vLLM engine, with KV caches transferred between pools via NIXL, and all workloads tested under measured saturation conditions.

Across three mixed, bursty arrival traces, the calibrated router achieved a peak mean goodput of 0.864, while round-robin, least-load, and length-heuristic methods ranged from 0.835 to 0.847. More importantly, the calibrated router had the lowest variance across traces, meaning its performance is more stable and predictable.

Looking at individual traces: it outperformed round-robin and the length heuristic on all three, and beat the least-load strategy on two of them. Only on the third trace did it fall behind least-load — by a margin of just 0.003, well within run-to-run noise.

Goodput, as used in this paper, is a key metric for measuring LLM serving quality. It is defined as the proportion of requests — or the throughput — successfully completed while satisfying SLO (Service Level Objective) constraints. Unlike raw throughput, goodput treats any request that exceeds its latency budget as a failure and excludes it from the count, thereby capturing both processing speed and service quality compliance. NIXL is NVIDIA's low-latency KV cache transfer library, designed specifically for disaggregated inference scenarios. It efficiently moves KV caches from prefill instances to decode instances after prefill computation completes; transfer latency directly affects end-to-end request completion time. In disaggregated architectures, KV cache transfer overhead is a significant performance bottleneck and a cost dimension that routing decisions must account for.

Why Hardware Calibration Is Critical

The most thought-provoking finding of this research is: hardware calibration makes or breaks the approach.

Using constants derived directly from the simulator without real-hardware calibration causes the router to lose 4.5 goodput points and roughly 40% of its tail-latency advantage. At that point, the cost scorer degrades to little more than a queue counter — meaning an uncalibrated intelligent router is functionally no different from the most naive load balancing.

This finding has direct implications for engineering practice: the value of a learned router lies not in how sophisticated its algorithm is, but in how accurately it reflects the true behavior of the underlying hardware. Theoretical optimality derived without hardware grounding will be significantly diminished in production.

Where It Helps and Where It Doesn't

The paper honestly delineates the boundaries of this approach — which is more useful than simply promoting its advantages.

Gains grow with scale and heterogeneity: the larger the decode pool and the more heterogeneous the traffic, the more pronounced the calibrated router's advantage. But in small pools with only three instances, the gains essentially disappear — because simple queue counting is often good enough there.

Extreme scarcity causes failure: under severe resource contention, the greedy cost-minimization strategy piles requests onto the instance with the lowest score (cheapest), which actually performs worse than blind spreading. This is a reminder that cost-driven greedy strategies can fail counterintuitively at the edge of saturation.

Traffic heterogeneity here refers to the degree of variation among concurrent requests in terms of input length, expected output length, and SLO requirements. When all requests are highly similar, simple round-robin already achieves near-uniform load distribution and the marginal value of intelligent routing is limited. When requests of varying lengths and SLO strictness arrive simultaneously, queue states across instances can diverge rapidly due to chance allocation differences — and that's when accurate cost modeling pays off, properly interleaving high-cost and low-cost requests to prevent cascading pile-ups. This also explains why the paper chose bursty arrival traces as the test scenario: bursty traffic naturally creates load imbalance, providing the conditions most likely to amplify differences between routing strategies.

The Most Practical Payoff: Saving One GPU

For cost-sensitive deployments, the paper delivers a compelling conclusion: with a properly calibrated cost model, the learned router needs only six GPUs to match the goodput that a round-robin strategy requires seven GPUs to achieve.

Translated into production terms, this means approximately 14% fewer GPUs for equivalent service quality. For large-scale LLM serving deployments that routinely operate hundreds or thousands of GPUs, this kind of hardware savings driven purely by smarter software routing represents real, tangible cost optimization.

Conclusion

The significance of this research goes beyond proposing a better routing algorithm. It uses rigorous empirical methods to reveal a practical engineering principle: calibrate first, then route. The gains from intelligent routing are real — but they are highly dependent on accurate hardware modeling, and only pay off under specific scale and traffic conditions. For teams building or optimizing disaggregated LLM serving systems, this measured and grounded study offers a rare and valuable practical guide.