Unverified50% confidenceBenchmarkExact time
Loading a 68GB MoE into 12GB VRAM + 16GB RAM in standard llama.cpp produces about 1500 major page faults per token, and generation speed is only 1.47 tok/s.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/31/2026
First Seen
Valid until: 1/29/2027
Sources
Running a 68GB Large Model on a 12GB GPU: Betelgeuse Engine Benchmarked at 22 tok/s
redditr/LocalLLaMA10/31/2026
Related Entities
Related Claims
UnverifiedRunning Large 4 on a typical laptop with 16GB/32GB of memory generates at about 15 tokens per second.73% similarUnverifiedThe PagedAttention paper uses OPT-13B in FP16 as an example, where each token requires approximately 800 KB, and 2048 tokens require approximately 1.6 GB of KV Cache.72% similarUnverifiedThe diagnostic signature of VRAM jitter is PCIe reads spiking from a normal ~1.5 GB/s to 9–13 GB/s, while GPU power consumption drops from 450 W to 340 W.72% similarUnverifiedRunning a 16-bit precision 70B model on 2×80GB GPUs leaves only 2.9 GB of VRAM after the weights are loaded, accommodating only about 9,000 in-flight tokens, which in practice cannot serve requests effectively.71% similarVerifiedTaking a 70B parameter model with FP16 precision and 32 attention head layers as an example, the KV Cache for a single 200K-length request requires approximately 80–120 GB of VRAM, far exceeding the 80 GB VRAM limit of a single A100 GPU.70% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1120763API
curl https://kongchang.com/api/v1/knowledge/claims/1120763MCP
get_claim(id=1120763)