Llama-3.3-70B
An open-source large language model released by Meta with 70B parameters. It belongs to the Llama 3 series and supports multiple languages and long context.
Core Facts
Timeline (last 90 days)
When deploying Llama-3.3-70B (bf16) with TP=4 on four cards and keeping the default max-model-len, worker nodes OOM under real long-request traffic of 6k to 8k tokens, and GPUStack promptly restarts them.
Fixing max-model-len at 16384 halves the KV Cache upper bound; the original team's measured real prompt p99 was 11k tokens, so 16k both leaves headroom and avoids doubling the pool.
After adopting GQA, Llama 3.3 70B has a KV Cache of approximately 320 KB per token.
All Facts (4)
以LLaMA-3 70B（80层、64头、每头128维）为例，单个token的KV Cache约占2.6MB（FP16精度），32K上下文的KV Cache总量约达83GB
80%UnverifiedFixing max-model-len at 16384 halves the KV Cache upper bound; the original team's measured real prompt p99 was 11k tokens, so 16k both leaves headroom and avoids doubling the pool.
50%UnverifiedAfter adopting GQA, Llama 3.3 70B has a KV Cache of approximately 320 KB per token.
50%UnverifiedWhen deploying Llama-3.3-70B (bf16) with TP=4 on four cards and keeping the default max-model-len, worker nodes OOM under real long-request traffic of 6k to 8k tokens, and GPUStack promptly restarts them.
50%