Unverified50% confidenceBenchmarkExact time
On the Q8_K_XL quantization at about 190GB total size, bel-server achieves 8 tok/s cold and 12 tok/s warm (peaking at 21 tok/s) on an RTX 3060 12GB + 16GB DDR4 machine, whereas stock llama.cpp immediately OOMs or crawls at <0.1 tok/s.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/31/2026
First Seen
Valid until: 1/29/2027
Sources
Running a 68GB Large Model on a 12GB GPU: Betelgeuse Engine Benchmarked at 22 tok/s
redditr/LocalLLaMA10/31/2026
Related Entities
Related Claims
UnverifiedA ~25 GB Q3 model running entirely within the RTX 4090's VRAM (bandwidth ~1 TB/s) theoretically achieves ~40 tokens per second, whereas running it in DDR5 memory (~80 GB/s) drops throughput to 1–3 tokens per second.74% similarUnverifiedIn single-user scenarios llama.cpp runs an 8B model at about 115 tok/s with about 12ms first-token latency on an RTX 4090, but at 32 concurrency throughput is only about 210 tok/s.73% similarUnverifiedIn the 12GB VRAM configuration, Breeze TTS2 requires about 8.5GB of VRAM (weights plus runtime overhead) and must run entirely on the GPU, while the LLM portion takes about nearly 4GB at a 128K context.72% similarUnverifiedA UP host running MiniCPM 2B with an RTX 4080S (16GB VRAM) + 64GB RAM configuration achieved an inference speed of approximately 110 tokens/second in real-world testing72% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1120770API
curl https://kongchang.com/api/v1/knowledge/claims/1120770MCP
get_claim(id=1120770)