Concept分层KV缓存 / 分层缓存

Tiered KV Cache

A multi-tier KV cache architecture used in LLM inference systems, typically comprising GPU memory (HBM), CPU memory (DRAM), and NVMe SSD storage tiers. These tiers differ significantly in access latency, capacity, and cost.

Timeline (last 90 days)

Oct 6

The tiered cache system in vLLM is commonly referred to as Tiered KV Cache and can span multiple storage tiers including GPU HBM, CPU DRAM, NVMe SSD, or remote distributed caches.

Unverified50%

All Facts (1)

Unverified

The tiered cache system in vLLM is commonly referred to as Tiered KV Cache and can span multiple storage tiers including GPU HBM, CPU DRAM, NVMe SSD, or remote distributed caches.

50%

Source Articles

vLLM v0.31.1rc0 Released: Exposing Cached Prompt Token Metrics by Cache TierOct 6