Unverified50% confidenceSolutionExact time
Diffbot's approach moves 4,608 of the 15,360 routed experts, plus lookup tables (Engram tables int4 and embedding), entirely to host memory, with the GPU reading them over PCIe via UVA, reducing the footprint per card to 72.4 GiB.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/16/2026
First Seen
Valid until: 1/14/2027
Sources
Related Entities
Related Claims
UnverifiedCompressing the context from 128K to 32K may quadruple the number of concurrent users a single GPU can serve, representing a trade-off between throughput and context length.64% similarVerifiedTaking a 70B parameter model with FP16 precision and 32 attention head layers as an example, the KV Cache for a single 200K-length request requires approximately 80–120 GB of VRAM, far exceeding the 80 GB VRAM limit of a single A100 GPU.64% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1044812API
curl https://kongchang.com/api/v1/knowledge/claims/1044812MCP
get_claim(id=1044812)