Conceptincremental publication
Incremental Publishing
A cache-page publishing strategy in prefix caching for LLM inference in which newly completed cache pages are published immediately after each batched forward pass, allowing pages to be reused by other requests while generation is still in progress.
Timeline (last 90 days)
Oct 16
In mini-vllm-rs, after introducing incremental publishing, each follower request recovers 480 of 493 prompt tokens.
Unverified50%
Oct 16
In the groups of 2, 4 and 8 concurrent requests, compared with 'publish only when complete', incremental publishing reduced average TTFT by 48–93%.
Unverified50%