Cache-Aware Routing
A routing strategy in LLM inference serving that preferentially sends requests to replicas already holding the same prefix in cache, reusing the KV cache to reduce prefill computation. The trade-off is potentially increased queueing latency.
Timeline (last 90 days)
llm-d 1.0 introduces cache-aware routing, which incorporates cache state into routing decisions and directs requests to backend instances that already hold the relevant context.
Cache-aware routing preferentially sends requests to replicas that already have the same prefix in cache; when the cache is hit, prefill is almost zero, and first-token latency drops significantly.
All Facts (2)
llm-d 1.0 introduces cache-aware routing, which incorporates cache state into routing decisions and directs requests to backend instances that already hold the relevant context.
50%UnverifiedCache-aware routing preferentially sends requests to replicas that already have the same prefix in cache; when the cache is hit, prefill is almost zero, and first-token latency drops significantly.
50%