KV感知路由
一种LLM推理请求路由策略，将请求导向已缓存相关上下文的节点，从而在多轮AI工作流中维持较高的KV缓存复用率，减少重复prefill带来的算力浪费，是分离式Prefill/Decode架构的关键配套组件。
Core Facts
Timeline (last 90 days)
By offloading output processing to a Sidecar and using a Rust-written frontend, latency can be significantly reduced; KV-aware routing can direct requests to more suitable instances based on cache state, improving cache hit rates and reducing redundant computation.
KV-aware routing directs requests to nodes with higher cache hit rates by determining which workers have already cached the relevant KV blocks, thereby reducing the computation in the prefill stage.
Conventional KV caching is an optimization within a single request: within the same conversation round, the KV generated from historical messages does not need to be recomputed, and only the newly input content is computed incrementally.
All Facts (3)
Conventional KV caching is an optimization within a single request: within the same conversation round, the KV generated from historical messages does not need to be recomputed, and only the newly input content is computed incrementally.
65%UnverifiedBy offloading output processing to a Sidecar and using a Rust-written frontend, latency can be significantly reduced; KV-aware routing can direct requests to more suitable instances based on cache state, improving cache hit rates and reducing redundant computation.
50%UnverifiedKV-aware routing directs requests to nodes with higher cache hit rates by determining which workers have already cached the relevant KV blocks, thereby reducing the computation in the prefill stage.
50%