Llama-2-7B
The 7-billion-parameter variant in Meta's Llama 2 model series, trained with a deliberate over-training strategy, achieving a token-to-parameter ratio of approximately 290:1.
Core Facts
以LLaMA-3 70B（80层、64头、每头128维）为例，单个token的KV Cache约占2.6MB（FP16精度），32K上下文的KV Cache总量约达83GB
For a typical 7B model with 32 layers and 32 attention heads, storing the KV Cache for each token requires approximately 0.5 MB of VRAM, so generating 1,000 tokens requires an additional ~500 MB of VRAM
Timeline (last 90 days)
An M2 Max Mac Studio can run Llama-2-7B inference at 8 tokens per second.
Under the same average activated parameter budget, LRCC improves average downstream task accuracy by 7.6 percentage points on Llama-2-7B compared to static methods.
The token-to-parameter ratios of Llama-2-7B and Gemma-2-9B are approximately 290 and 889, respectively, far exceeding Chinchilla's 20:1.
All Facts (5)
以LLaMA-3 70B（80层、64头、每头128维）为例，单个token的KV Cache约占2.6MB（FP16精度），32K上下文的KV Cache总量约达83GB
80%VerifiedFor a typical 7B model with 32 layers and 32 attention heads, storing the KV Cache for each token requires approximately 0.5 MB of VRAM, so generating 1,000 tokens requires an additional ~500 MB of VRAM
70%UnverifiedThe token-to-parameter ratios of Llama-2-7B and Gemma-2-9B are approximately 290 and 889, respectively, far exceeding Chinchilla's 20:1.
85%UnverifiedAn M2 Max Mac Studio can run Llama-2-7B inference at 8 tokens per second.
50%UnverifiedUnder the same average activated parameter budget, LRCC improves average downstream task accuracy by 7.6 percentage points on Llama-2-7B compared to static methods.
50%