FluidPD: In-Place Elastic Scheduling Tackles the P/D Resource Imbalance Problem in LLM Inference

The inference architecture for large language model (LLM) serving is undergoing a wave of fine-grained optimization. The Prefill-Decode (P/D) disaggregated architecture has become increasingly mainstream, as it enables independent handling of two execution phases with fundamentally different characteristics. But this architecture has a long-standing pain point: when real-world workload demand ratios fluctuate dramatically, fixed resource allocations quickly become mismatched, causing latency SLO (Service Level Objective) violations. A new paper on arXiv titled FluidPD proposes an "in-place elasticity" solution specifically targeting this problem.

The Structural Dilemma of P/D Disaggregation

To understand the value of FluidPD, we first need to understand the fundamental differences between the Prefill and Decode phases. The Prefill phase processes the user's input prompt — it is compute-intensive, consuming a large batch of tokens at once. The Decode phase generates output token by token — it is memory-bandwidth-intensive and far more sensitive to latency. Because the two phases have such different execution characteristics and SLO targets, splitting them across separate workers for independent optimization is a natural way to improve overall service quality.

The problem lies in the fixed allocation ratio. Existing systems typically use a fixed prefill/decode worker ratio combined with request routing to distribute load. But as the paper points out, real-world workloads exhibit both short bursts and sustained shifts in demand ratios. This means a well-calibrated resource allocation at one moment can become severely imbalanced the next — and even when spare compute exists elsewhere in the system, latency SLOs still get violated.

KV Cache is another critical piece of context for understanding the P/D disaggregated architecture. After the Prefill phase completes its computation, it produces a set of intermediate state matrices — the Key and Value tensors used in the Attention mechanism — which must be continuously read throughout the Decode phase. In a disaggregated setup, this means KV Cache must be transferred between prefill workers and decode workers, introducing significant network bandwidth overhead. When worker roles are dynamically adjusted, efficiently managing and migrating KV Cache becomes a core engineering challenge that systems like FluidPD must address. This "compute in prefill, read in decode" asymmetry further amplifies the resource characteristic differences between the two phases, making the fixed-ratio problem even more pronounced.

Why Traditional Autoscaling Falls Short

The most intuitive response to imbalance is autoscaling. But the paper explicitly identifies three shortcomings of this approach: slow reaction time, reliance on idle GPU reserves, and inability to directly address short-timescale phase imbalances.

Autoscaling fundamentally means "adding capacity" — it requires scheduling new GPUs, loading models, and starting inference engines, a process that routinely takes seconds to tens of seconds. For demand spikes operating at second or sub-second timescales, this response speed is far too slow. More critically, autoscaling depends on having idle GPU resources available to allocate, a condition that often doesn't hold in resource-constrained production environments. In other words, traditional autoscaling solves the problem of "insufficient total capacity," while the real pain point in P/D architectures is structural mismatch — total capacity may be sufficient, but it's allocated in the wrong direction.

FluidPD's Two Core Mechanisms

FluidPD's approach is not to add resources, but to make existing resources fluid — enabling SLO-aware in-place elasticity. It introduces two complementary mechanisms that each address imbalances at different timescales.

FluidToken: Handling Instantaneous Imbalances

FluidToken targets short bursts. When decode workers have spare capacity, it offloads a bounded fraction of prefill computation to those decode workers. The keyword here is "bounded" — the offload amount is constrained to avoid degrading decode latency from excessive borrowing. This effectively provides temporary access to the other side's idle compute to fill instantaneous gaps, without changing the worker's assigned role.

FluidRole: Handling Sustained Imbalances

When the imbalance is not a brief spike but a sustained demand shift, FluidToken's temporary borrowing is no longer sufficient. FluidRole's approach is to reassign worker roles in-place — switching running workers between their prefill and decode roles. The most elegant aspect is that this switch happens in place, avoiding model reloading and engine restarts. This sidesteps the most time-consuming parts of traditional autoscaling, minimizing the cost of role transitions.

Pressure Indices: Proactive Intervention Before SLO Violations

Both mechanisms are guided by lightweight pressure indices. This represents an important design philosophy shift in FluidPD: rather than reacting passively after SLOs have already been violated, the system exposes resource pressure on both the prefill and decode sides to intervene before latency problems actually materialize.

This proactive scheduling allows the system to respond at the first signs of imbalance rather than scrambling to recover after the fact. The lightweight design also ensures that pressure monitoring itself doesn't become a new performance burden.

---

In LLM inference, SLOs typically revolve around two core metrics: TTFT (Time To First Token) and TBT/TPOT (Token Between Tokens / Time Per Output Token). TTFT is primarily affected by the Prefill phase and reflects how quickly users start seeing a response after submitting a request; TBT is primarily affected by the Decode phase and determines the overall smoothness of generation. A core motivation for the P/D disaggregated architecture is that optimizing for these two metrics creates conflicting pressures — aggressively batching prefill improves throughput but increases TTFT, while protecting TTFT constrains prefill batch sizes. FluidPD's pressure indices must simultaneously sense SLO pressure on both sides, so their design must be able to distinguish between these two types of latency targets in order to make correct resource scheduling decisions.

---

Experimental Results and Significance

The paper evaluates FluidPD on production-level Azure trace workloads. Results show that compared to a statically configured SGLang baseline, FluidPD improves overall SLO attainment by up to 94.6 percentage points — a substantial number that demonstrates SLO-aware in-place P/D elasticity can significantly improve service quality without adding any additional workers.

From an industry perspective, FluidPD's significance goes beyond a single performance figure. It offers an alternative to simply "throwing more hardware at the problem." As LLM inference costs become a central concern for enterprise deployments, extracting higher service quality from a fixed GPU budget is increasingly the frontier of systems engineering. FluidPD's principle of "making resources fluid rather than adding resources" offers a compelling direction for engineering optimization of P/D disaggregated architectures.

It's worth noting that this is a freshly published arXiv preprint, and the conclusions above are still pending broader peer review and real-world deployment validation. That said, the two-tier elasticity mechanism and proactive pressure-based scheduling it proposes genuinely address real pain points in current LLM serving architectures.