Unverified50% confidenceFactExact time
A user runs the 397B model on 4× ASUS GB10 units achieving 30 tokens/s at only 400W of power consumption, compared to 50–60 tokens/s at 6kW previously.
1
Sources
50%
Confidence
Long-term
Relevance
10/4/2026
First Seen
Sources
Related Entities
Related Claims
VerifiedTaking LLaMA-2 70B as an example, when processing a 4096-token context with a batch size of 32, the KV Cache can consume over 80 GB of memory.74% similarUnverifiedTaking Llama-3 70B as an example, the prefill stage for processing 32K tokens on an A100 80GB GPU takes approximately 800–1200ms, while processing 2K tokens takes only about 50ms.71% similarUnverifiedQuantprobe claims to enable a machine with only 6GB of VRAM and 16GB of RAM to run 30B-class models at 22 tokens/s.71% similarUnverifiedUnder the M5 Ultra's 1.2 TB/s bandwidth, the theoretical speed of a 100 GB quantized model is approximately 12 tokens/s, a 200 GB model approximately 6 tokens/s, and a 300 GB model approximately 4 tokens/s.70% similarUnverifiedIn a controlled test, the 35B model ran at approximately 100 tokens/second (Mac Studio), roughly five times the 9B model's 16 tokens/second70% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/974692API
curl https://kongchang.com/api/v1/knowledge/claims/974692MCP
get_claim(id=974692)