Llama 3.1 8B
The 8-billion-parameter version of Meta's open-source large language model Llama 3.1, commonly used in research and resource-constrained settings.
Core Facts
以LLaMA-3 70B（80层、64头、每头128维）为例，单个token的KV Cache约占2.6MB（FP16精度），32K上下文的KV Cache总量约达83GB
Even compressing the weights to 214GB using 4-bit quantization still far exceeds the capacity of a single 48GB GPU.
Llama 3 8B has 32 layers, 8 KV heads (GQA architecture with 32 query heads), a head dimension of 128, and a KV cache of approximately 1.07 GB for a single sequence under FP16 with an 8192 context length.
Timeline (last 90 days)
The token-probability scoring method significantly outperforms direct prompt-based classification, with particularly notable gains on the 8B model.
The study evaluated the token-probability scoring method using Llama 3.1 8B and 70B models on three datasets: TOEFL9, Write and Improve 2024, and BULATS-L1.
Llama 3.1 8B has 8,006,464,512 parameters (according to Meta's official model card).
Grouped-query attention (GQA) was proposed by Ainslie et al. in 2023 and is a compromise between multi-head attention (MHA) and multi-query attention (MQA); Llama 3.1 8B uses 32 query heads divided into 8 groups, with each group of 4 query heads sharing 1 KV head, shrinking the KV cache by 4x compared to MHA.
On the cluster panel, replacing the model's structured decision-making with code (based on the same extracted facts) improved family accuracy by an average of 20.5 percentage points, with Llama 3.1 8B rising from 1% under self-judgment to 56% under code adjudication.
Groq measured inference speed on Llama 3.1 8B exceeds 500 tokens/second
Llama 3.1 8B and Qwen2.5 14B are recommended for general reasoning and Q&A, with the latter performing better on Chinese and complex reasoning.
The KV cache overhead per token for Llama 3.1 8B is approximately 128 KB.
The core idea of model routing is that not all tasks require the most powerful model; simple tasks can use lightweight models such as GPT-4o-mini, Claude 3 Haiku, or Llama 3.1 8B, potentially at just 1/50 or even 1/100 the cost of flagship models.
Llama 3 8B has 32 layers, 8 KV heads (GQA architecture with 32 query heads), a head dimension of 128, and a KV cache of approximately 1.07 GB for a single sequence under FP16 with an 8192 context length.
All Facts (12)
以LLaMA-3 70B（80层、64头、每头128维）为例，单个token的KV Cache约占2.6MB（FP16精度），32K上下文的KV Cache总量约达83GB
80%VerifiedEven compressing the weights to 214GB using 4-bit quantization still far exceeds the capacity of a single 48GB GPU.
70%VerifiedLlama 3 8B has 32 layers, 8 KV heads (GQA architecture with 32 query heads), a head dimension of 128, and a KV cache of approximately 1.07 GB for a single sequence under FP16 with an 8192 context length.
65%UnverifiedThe token-probability scoring method significantly outperforms direct prompt-based classification, with particularly notable gains on the 8B model.
50%UnverifiedThe study evaluated the token-probability scoring method using Llama 3.1 8B and 70B models on three datasets: TOEFL9, Write and Improve 2024, and BULATS-L1.
50%UnverifiedLlama 3.1 8B has 8,006,464,512 parameters (according to Meta's official model card).
50%UnverifiedGrouped-query attention (GQA) was proposed by Ainslie et al. in 2023 and is a compromise between multi-head attention (MHA) and multi-query attention (MQA); Llama 3.1 8B uses 32 query heads divided into 8 groups, with each group of 4 query heads sharing 1 KV head, shrinking the KV cache by 4x compared to MHA.
50%UnverifiedOn the cluster panel, replacing the model's structured decision-making with code (based on the same extracted facts) improved family accuracy by an average of 20.5 percentage points, with Llama 3.1 8B rising from 1% under self-judgment to 56% under code adjudication.
50%UnverifiedGroq measured inference speed on Llama 3.1 8B exceeds 500 tokens/second
50%UnverifiedLlama 3.1 8B and Qwen2.5 14B are recommended for general reasoning and Q&A, with the latter performing better on Chinese and complex reasoning.
50%UnverifiedThe KV cache overhead per token for Llama 3.1 8B is approximately 128 KB.
50%UnverifiedThe core idea of model routing is that not all tasks require the most powerful model; simple tasks can use lightweight models such as GPT-4o-mini, Claude 3 Haiku, or Llama 3.1 8B, potentially at just 1/50 or even 1/100 the cost of flagship models.
50%