Unverified50% confidenceBenchmarkExact time
Prefill-Decode (PD) disaggregation requires transferring KV Cache from the Prefill instance to the Decode instance; a CX7 NIC at full 400 Gbps IB delivers approximately 50 GB/s, and transferring 40 GB of KV Cache takes 0.8 seconds.
1
Sources
50%
Confidence
Long-term
Relevance
9/23/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedThe deployment uses a llama.cpp server with 4-bit KV cache, context length set to 100,000, 32 layers offloaded to system memory, and lazy mode enabled to force the Ngram table to be read from SSD.69% similarUnverifiedThe FP4-based model (svdq-fp4_r32) built on Nunchaku has a size of 6.6GB, fits entirely within 8GB of VRAM without CPU offloading, and reduces the per-step time to 3.4–3.8 seconds.69% similarUnverifiedFor the LLaMA-2 70B model, when processing a context of 4096 tokens, the KV Cache requires approximately 20 GB of system memory at FP16 precision.68% similarUnverifiedFor a 32-layer, 4096-dimensional model, the KV Cache for processing 2048 tokens occupies approximately 1 GB of VRAM68% similarUnverifiedRunning complex Agent workflows is recommended on dual 4090 or 5090 GPUs, as the space left for the KV Cache after model weights occupy a single card with 32GB of VRAM is limited.67% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/942624API
curl https://kongchang.com/api/v1/knowledge/claims/942624MCP
get_claim(id=942624)