Unverified50% confidenceBenchmarkExact time
Under the same conditions of Llama-3.1-8B, batch size 32, and context 2048, vLLM achieves about 450 tok/s, llama.cpp about 280 tok/s, and Ollama about 120 tok/s.
1
Sources
50%
Confidence
Short-term (~14 days)
Relevance
10/12/2026
First Seen
Valid until: 10/26/2026
Sources
Related Entities
Related Claims
UnverifiedLlama 3 8B has 32 layers, 8 KV heads (GQA architecture with 32 query heads), a head dimension of 128, and a KV cache of approximately 1.07 GB for a single sequence under FP16 with an 8192 context length.71% similarUnverifiedAfter adopting GQA, Llama 3.3 70B has a KV Cache of approximately 320 KB per token.70% similarUnverifiedFull fine-tuning of Llama-3 (8B) requires approximately 80 GB or more of VRAM, while Llama-3 (70B) requires more than 400 GB.67% similarUnverifiedThe context length of mainstream open-source multimodal models is typically between 4K and 32K tokens: LLaVA-1.5 is 4,096 tokens, Qwen-VL is 8,192 tokens, and InternVL2 extends to 32K.67% similarUnverifiedThe KV cache overhead per token for Llama 3.1 8B is approximately 128 KB.67% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1008120API
curl https://kongchang.com/api/v1/knowledge/claims/1008120MCP
get_claim(id=1008120)