Prefix Sharing
A large-model inference optimization technique that allows multiple requests to share the KV cache of identical prefixes, avoiding redundant computation and improving GPU utilization.
Core Facts
Timeline (last 90 days)
The prefix-sharing mechanism relies on a global cache pool indexed by content hashes or prefix fingerprints; if the index keys are too coarse-grained or cache blocks are not invalidated in time, subsequent requests may read Key/Value tensors left over from a previous request.
Methods for compressing the KV Cache include quantization, low-rank approximation, and evicting old tokens by importance; their direct effect is to increase the number of tokens (capacity) that a unit of GPU memory can hold, rather than to save compute or transfer.
All Facts (2)
Methods for compressing the KV Cache include quantization, low-rank approximation, and evicting old tokens by importance; their direct effect is to increase the number of tokens (capacity) that a unit of GPU memory can hold, rather than to save compute or transfer.
65%UnverifiedThe prefix-sharing mechanism relies on a global cache pool indexed by content hashes or prefix fingerprints; if the index keys are too coarse-grained or cache blocks are not invalidated in time, subsequent requests may read Key/Value tensors left over from a previous request.
50%