Unverified50% confidenceFactExact time
The tiered cache system in vLLM is commonly referred to as Tiered KV Cache and can span multiple storage tiers including GPU HBM, CPU DRAM, NVMe SSD, or remote distributed caches.
1
Sources
50%
Confidence
Long-term
Relevance
10/6/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedA tiered memory architecture organizes HBM, DRAM, and CXL extended memory pools into a hierarchical structure, with HBM serving as the high-speed cache layer, DRAM as the main memory layer, and CXL providing large-capacity shareable memory resources.74% similarUnverifiedvLLM v0.28.0's tiered KV offloading adds a disk tier and supports out-of-tree secondary cache layers.71% similarUnverifiedvLLM引入PagedAttention技术将KV Cache分割为固定大小物理块按需分配，而llama.cpp采用预分配策略在模型加载时一次性保留完整KV Cache空间70% similarUnverifiedFP8 KV cache is a quantization technique that compresses the Key-Value cache in the attention mechanism from FP16 or BF16 to 8 bits, which can halve the GPU memory usage of the KV cache70% similarUnverifiedThe core bottleneck in large model inference on GPUs is memory bandwidth; autoregressive generation of each token requires loading the full KV Cache from HBM, with compute utilization typically below 5%.70% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/982067API
curl https://kongchang.com/api/v1/knowledge/claims/982067MCP
get_claim(id=982067)