Unverified50% confidenceTradeoffExact time
The CPU offloading strategy keeps the weights of inactive experts resident in system memory and only temporarily loads them into GPU VRAM when selected by the router, at the cost of introducing PCIe bus transfer latency.
1
Sources
50%
Confidence
Long-term
Relevance
10/4/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedKeeping model weights fully resident in GPU memory avoids memory swap-in/swap-out overhead, whereas models that exceed GPU memory capacity require data to be moved between CPU and GPU, causing latency to rise significantly.78% similarUnverifiedGPU 推理速度通常是纯 CPU 的 5-20 倍，当模型太大无法放入显存时部分层会卸载到内存由 CPU 处理（混合运行/partial offload）75% similarUnverifiedCPU offload places some Transformer layers in system main memory to be computed by the CPU, with data transferred over the PCIe bus (PCIe 4.0 x16 theoretical bandwidth is approximately 32 GB/s), while GPU memory bandwidth is typically in the range of hundreds to over a thousand GB/s, making the bandwidth gap the fundamental cause of the performance bottleneck.74% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/974638API
curl https://kongchang.com/api/v1/knowledge/claims/974638MCP
get_claim(id=974638)