Vesta
An inference scheduling study published on arXiv that proposes decoupling session-level KV cache ownership from request-level execution placement. It uses queueing theory to make a quantifiable trade-off between cache locality and queue congestion, thereby optimizing throughput and time to first token in multi-turn session scenarios.
Timeline (last 90 days)
The arXiv paper Vesta proposes decoupling 'session-level KV cache affinity' from 'request-level execution placement', using queuing theory to trade off between cache locality and queue congestion.
Vesta improves throughput by 14% and reduces average TTFT by 57.8% under co-located deployment, and reduces P99 TTFT by up to 29% under disaggregated deployment.
All Facts (2)
The arXiv paper Vesta proposes decoupling 'session-level KV cache affinity' from 'request-level execution placement', using queuing theory to trade off between cache locality and queue congestion.
50%UnverifiedVesta improves throughput by 14% and reduces average TTFT by 57.8% under co-located deployment, and reduces P99 TTFT by up to 29% under disaggregated deployment.
50%