Unverified50% confidenceSolutionExact time
On heterogeneous nodes where TP cannot be used, GPUStack falls back to llama-box with layer offloading; num_gpu 60 (out of 80 layers) is the honest upper limit on 48 GB of VRAM, and setting it to 80 crashes on the first 4k prompt.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/18/2026
First Seen
Valid until: 1/16/2027
Sources
Related Entities
Related Claims
UnverifiedWhen deploying Llama-3.3-70B (bf16) with TP=4 on four cards and keeping the default max-model-len, worker nodes OOM under real long-request traffic of 6k to 8k tokens, and GPUStack promptly restarts them.68% similarUnverifiedRunning a 16-bit precision 70B model on 2×80GB GPUs leaves only 2.9 GB of VRAM after the weights are loaded, accommodating only about 9,000 in-flight tokens, which in practice cannot serve requests effectively.67% similarUnverified1664x960 occupies 21.6GB in 24GB of VRAM (1.2GB baseline) and can only run when the GPU is truly idle; multiple rendering failures occurred because llama-swap and a resident engine slipped back onto the GPU mid-render.66% similarUnverifiedLlama 3 8B has 32 layers, 8 KV heads (GQA architecture with 32 query heads), a head dimension of 128, and a KV cache of approximately 1.07 GB for a single sequence under FP16 with an 8192 context length.66% similarVerifiedThe consumer-grade GPU RTX 4090 has a VRAM ceiling of 24 GB.65% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1063645API
curl https://kongchang.com/api/v1/knowledge/claims/1063645MCP
get_claim(id=1063645)