GPT-2 small
The small version in the GPT-2 series released by OpenAI, with approximately 124 million parameters, and one of the representative early open language models.
Timeline (last 90 days)
15M parameters is roughly equivalent to about one-eighth of early GPT-2 small (about 117M parameters).
The experiment used pretrained GPT-2 small (124M parameters), an NVIDIA A40 GPU, fp32 precision with TF32 disabled, batch size 1, and greedy decoding.
One developer implemented the KV Cache for GPT-2 small from scratch in PyTorch and measured three sets of data: correctness, latency, and memory.
The author's GPT-2 small KV cache implementation code and article are publicly available on a personal blog and GitHub repository (angry-kratos.github.io), and the same set of numbers can be reproduced.
Under the same experimental setup, the total time to generate 1016 tokens drops from 14.66 seconds to 5.03 seconds.
On a single NVIDIA A40 with fp32 and TF32 disabled, batch size 1 and greedy decoding, after GPT-2 small uses the KV cache, single-token latency at context length 1000 drops from 29.69 ms to 4.95 ms, about 6x.
Under the same experimental setup, at context length 1000 the estimated FLOPs drop by about 1000x, while single-token latency drops only about 6x.
Keera's architecture is based on GPT-2 small: 12 Transformer blocks, hidden dimension 768, and a total parameter count of 97.5M.
机制可解释性入门可从小模型（如 GPT-2 small）的可解释性分析做起，再逐步挑战更复杂的机制
All Facts (9)
Under the same experimental setup, the total time to generate 1016 tokens drops from 14.66 seconds to 5.03 seconds.
70%Unverified15M parameters is roughly equivalent to about one-eighth of early GPT-2 small (about 117M parameters).
50%UnverifiedThe experiment used pretrained GPT-2 small (124M parameters), an NVIDIA A40 GPU, fp32 precision with TF32 disabled, batch size 1, and greedy decoding.
50%UnverifiedUnder the same experimental setup, at context length 1000 the estimated FLOPs drop by about 1000x, while single-token latency drops only about 6x.
50%UnverifiedOn a single NVIDIA A40 with fp32 and TF32 disabled, batch size 1 and greedy decoding, after GPT-2 small uses the KV cache, single-token latency at context length 1000 drops from 29.69 ms to 4.95 ms, about 6x.
50%UnverifiedThe author's GPT-2 small KV cache implementation code and article are publicly available on a personal blog and GitHub repository (angry-kratos.github.io), and the same set of numbers can be reproduced.
50%UnverifiedKeera's architecture is based on GPT-2 small: 12 Transformer blocks, hidden dimension 768, and a total parameter count of 97.5M.
50%Unverified机制可解释性入门可从小模型（如 GPT-2 small）的可解释性分析做起，再逐步挑战更复杂的机制
50%UnverifiedOne developer implemented the KV Cache for GPT-2 small from scratch in PyTorch and measured three sets of data: correctness, latency, and memory.
50%