SWA Bounded Replay
A cache optimization technique in DeepInfer that reduces the prefill computation for long contexts by limiting the replay window.
Timeline (last 90 days)
DeepInfer uses SWA bounded replay technology to reduce the last 2,560 tokens recomputed when rebuilding the cache to 128, cutting total prefill computation nearly in half and decoupling long-context cost from length.
This time vLLM reduced the compute and memory access overhead of the prefill stage through means such as SWA bounded replay and kernel fusion.
All Facts (2)
DeepInfer uses SWA bounded replay technology to reduce the last 2,560 tokens recomputed when rebuilding the cache to 128, cutting total prefill computation nearly in half and decoupling long-context cost from length.
50%UnverifiedThis time vLLM reduced the compute and memory access overhead of the prefill stage through means such as SWA bounded replay and kernel fusion.
50%