AKV
A serving engine designed for cache engineering that internally manages stateful KV cache reuse, eviction, and restoration, while exposing a stateless interface externally to remain compatible with existing inference serving APIs.
Timeline (last 90 days)
The paper's authors designed the AKV serving engine, which internally manages stateful cache reuse, eviction, and restoration while exposing a stateless interface externally that is compatible with existing serving APIs.
On the AKV engine, output throughput improves by up to 1.2x and P95 request latency speeds up by 1.5x.
The open-source code of the AKV engine has been released on GitHub, providing a reproducible foundation for subsequent research.
All Facts (3)
The paper's authors designed the AKV serving engine, which internally manages stateful cache reuse, eviction, and restoration while exposing a stateless interface externally that is compatible with existing serving APIs.
50%UnverifiedOn the AKV engine, output throughput improves by up to 1.2x and P95 request latency speeds up by 1.5x.
50%UnverifiedThe open-source code of the AKV engine has been released on GitHub, providing a reproducible foundation for subsequent research.
50%