Unverified50% confidenceEventExact time
When deploying Llama-3.3-70B (bf16) with TP=4 on four cards and keeping the default max-model-len, worker nodes OOM under real long-request traffic of 6k to 8k tokens, and GPUStack promptly restarts them.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/18/2026
First Seen
Valid until: 1/16/2027
Sources
Related Entities
Related Claims
UnverifiedBased on public data for high power plus MLX or Ollama, the decoding speed of different models on M4 Max is roughly a ladder as follows: Nemotron 3 Nano about 160 tokens/s, Qwen 3.5 35B at 107 to 130 tokens/s, Gemma 4 26B about 75 tokens/s, DeepSeek V4 Flash about 27 tokens/s, and dense 27B about 22 tokens/s.72% similarUnverifiedThe bfloat16 baseline throughput is 18.8K tokens/s/GPU, while the fastest custom path reaches 37.9K tokens/s/GPU, nearly doubling the throughput.71% similarUnverifiedWith a 5070 12 GB GPU paired with 64 GB of memory, speed can reach 93 tokens/s in short chat scenarios and 74 tokens/s for 128K long contexts.69% similarUnverifiedThe M5 Max's token generation speed is approximately in the range of 10–30 tokens/s (depending on model size), while the NVIDIA H100's HBM3 memory bandwidth reaches up to 3.35 TB/s, enabling inference speeds tens of times faster than local devices.69% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1063641API
curl https://kongchang.com/api/v1/knowledge/claims/1063641MCP
get_claim(id=1063641)