MIT Author Builds a 2MB Model That Fits in CPU L3 Cache, Hits 3405 tok/s on a Single Core

The narrative around large model inference is almost entirely dominated by GPUs, but a Reddit post from developer mdnazib963 offers a different approach: instead of competing with GPUs on compute, make the model small enough to live inside the CPU's L3 cache. The result is 3405 tok/s on a single core of a $60 CPU, with a model that's just 1.9 MB.

A deliberately boring starting point

At the start of the year, the author set himself a spec: 50 tok/s on a single core. That number was deliberately chosen to be "boring" — llama.cpp can already do it on a laptop, so it's not much of a challenge. But he followed that thread for nine months, and what he ended up with was something completely different: a 1.9 MB model trained from scratch, running at 3405 tok/s on a single core of a $60 CPU.

What's really interesting isn't the number itself, but the one-sentence explanation: he redefined CPU cache from an "implementation detail" into a "design target."

One forward pass is one weight read

The math of inference is actually quite plain. Every forward pass has to read all the weights in full, and that read process is the cost itself.

The author's model has 1.91M parameters, which after int8 quantization take up 1.97 MB of weights. And the Ryzen 5 3400G he's using has 4 MB of L3 cache. 1.97 divided by 4 equals 49.8% — the weights take up exactly half of the L3.

If the weights fit in cache, decode degenerates into a cache streaming read: about 6.7 GB/s of sequential reads, served directly by L3. If they don't fit, every forward pass has to go to DRAM, and you end up wrestling with bandwidth, latency, and resource contention.

Most inference optimization is all about crunching compute, but the cache-fit approach goes the opposite way: by eliminating the memory problem, compute is no longer the bottleneck.

The model architecture isn't this small by accident — it was reverse-engineered from the cache: 4 gated short-conv blocks plus 2 grouped-query attention blocks (kv=1, so the KV cache is just one vector per layer instead of seq_len×d), a vocabulary of 4096 so the output head doesn't steal the show, and int8 export so the byte count equals the parameter count.

L3 Cache and the Memory Hierarchy: Modern CPUs have multiple levels of cache. L1 is the smallest and fastest (typically tens of KB, ~4-5 cycle latency), L2 is in the middle (hundreds of KB to a few MB), and L3 is the last-level cache, typically 4-32 MB, shared by all cores. The physical location of cache is right next to the compute units, so access latency is roughly one-hundredth of DRAM, and bandwidth is several times higher. "Cache-fit" means the model's entire weights reside permanently in L3, so every forward pass hits the cache when reading weights, bypassing DRAM's bandwidth bottleneck and access latency. Once L3 gets evicted by other data (like random reads from another process), the weights have to fall back to DRAM, and speed drops sharply — this is exactly the principle behind the evict_read mode described in the article. Understanding this hierarchy is the prerequisite for understanding "treating cache as a design target."

int8 Quantization and QAT-lite: int8 quantization compresses weights that would normally be 16-bit or 32-bit floats into 8-bit integers, so the byte count directly aligns with the parameter count (1.91M parameters ≈ 1.97 MB), making the weights easier to fit in cache. The cost is precision loss, which needs to be compensated for through quantization-aware training (QAT) that simulates quantization error during training. The "QAT-lite" mentioned in the article refers to a lightweight approach: not necessarily simulating throughout, but introducing quantization constraints in the final training phase or before export, keeping the perplexity difference between the C runtime and the PyTorch reference implementation within 0.082% (threshold ≤1%). per-input activation quantization is another piece of quantized inference: each input's activations get their own scaling factor, reducing error from dynamic range differences, at the cost of extra computation, which the author amortizes to an acceptable level with AVX2 instructions.

Proving the cache is actually doing something with a "control group"

A number without a control isn't really a measurement. So the author's benchmark includes an evict_read mode: while the model is decoding, another process hammers L3 with random reads.

State	tok/s
Warmed (weights resident in L3)	3405
Evicted (L3 polluted)	1681
Penalty	2.03×

Same binary, same model, same location — the only variable is whether the cache is resident. That 2.03× gap is measured, not argued. The author also provides a stream command as a reverse probe (128 MB sequential scan) to show your machine's raw DRAM bandwidth, so you can compare the two numbers yourself.

Grouped-Query Attention (GQA) and KV cache: In standard multi-head attention, the KV cache scales as seq_len × d × number of layers. The longer the sequence, the more it takes up, making it one of the main consumers of inference memory. GQA lets multiple query heads share a single set of key-value heads, compressing the number of KV heads from many to very few. The model in the article sets kv to 1, meaning each layer has just one key-value vector, so the KV cache becomes almost independent of sequence length. This both reduces memory footprint and lowers the read volume per decode step. For the cache-fit goal, if the attention layer's KV cache is too large, it will crowd out the already-tight L3 space, so kv=1 is a structurally necessary design to work with the cache, not just a speed optimization.

Speed wasn't traded for quality

Speed claims are cheap; claims of matching quality are expensive. The author's C runtime has to align with the PyTorch reference implementation it was exported from, and that's a hard threshold:

• Perplexity: C side 26.8407 vs PyTorch 26.8188, difference 0.082%, threshold is ≤1%
• Greedy token match: 99.41%, threshold is mismatch rate ≤1%
• Determinism tests and AddressSanitizer suite: all clean
• All threshold results are signed into the repo as JSON records in phase0/phase5_results.json

On the int8 implementation side, per-input activation quantization is amortized across all rows, the sign-extended dot product uses vpmovsxbw plus vpmaddwd and 4 accumulators, attention uses AVX2, and exp goes through a seven-term polynomial. Just swapping the float kernels for int kernels took it from 1494 tok/s to 3696 tok/s.

The whole pipeline is reproducible

The author lays out every step from data to running:

python data/clean.py && python data/generate.py && python data/pack.py
python train.py --train data/train.bin --val data/val.bin --out run --d 144 ...
python phase0_export.py --ckpt run/best.pt --out phase0/model.i8
phase0\build_p0.bat
phase0\phase0.exe phase0\model_p3.i8 bench data\val.bin 15.0 65

From Gutenberg corpus cleaning, synthetic task stream generation, BPE, training, QAT-lite int8 export, to the C runtime — the full chain, with corpus construction taking about 80 seconds. The benchmark prints a single line of JSON — either you reproduce his numbers or you don't.

What it can't do

The author is very straightforward about this part, and it's worth copying:

• The output is weak. 1.9M parameters is a design study, not a chatbot. It's a benchmark vehicle for validating the cache-fit argument, and the author makes no pretense that it has any other purpose.
• Currently Windows x64 only. The MSVC build works; Linux/macOS Makefiles are still in issue #2. If the post gets traction, this becomes top priority.
• One model, one class of machine. The numbers come from a Ryzen 5 3400G (4 MB L3). Bigger L3 means more headroom; the repo's info command will tell you what fit_half_l3 is for your CPU.

Sign-extended dot product and AVX2 implementation details: For int8 inference to be efficient on CPU, the key is using SIMD instructions to process multiple integers at once. The vpmovsxbw mentioned in the article sign-extends 8-bit integers to 16-bit, then vpmaddwd does multiply-add and accumulates to 32-bit, with 4 accumulators to hide instruction latency and improve pipeline throughput. The attention part is vectorized with AVX2, and exp uses a seven-term polynomial approximation to avoid calling expensive library functions. This kernel is the direct cause of the 1494→3696 tok/s improvement from swapping float for int: integer arithmetic is faster and more power-efficient than floating point on CPU, and it fits the rhythm of cache streaming reads better. To reproduce the numbers, these low-level implementations are an unavoidable step.

Why this is worth making public

The genuinely interesting claims in ML inference usually revolve around GPUs. But the memory hierarchy argument generalizes: if your weights can fit somewhere the compute unit can reach, the shape of the whole problem changes.

Aligning model size to cache is the cheapest place to validate this — you don't need an H100, a $60 old CPU is enough. The author's entire codebase, kernels, and measurements are all open source under MIT at GitHub's mdnazib963/CoreFlow, with weights released on Hugging Face at nazib61/coreflow-1.9m.

He also left a request: if you reproduce (or fail to reproduce) on your own hardware, open an issue with the JSON output. Hardware diversity is the whole point. This is actually closer to the essence of engineering spirit than the 3405 tok/s — the point of a number isn't to show off, but whether it holds up on someone else's machine.

One-line Summary

Developer fits a 1.9MB model into CPU L3 cache, hitting 3405 tok/s on a single core of a $60 CPU.

Paragraph Summary

The article introduces an unconventional inference experiment by developer mdnazib963: rather than competing with GPUs on compute, he designed the model's entire weights to reside in the CPU's L3 cache. With 1.91M parameters and int8 quantization, the weights take just 1.97 MB, exactly half of the Ryzen 5 3400G's 4 MB L3, so decode degenerates into a cache streaming read. Through an evict_read control experiment, the warmed state hits 3405 tok/s, dropping to 1681 tok/s once the cache is polluted — a 2.03× penalty proving the cache is doing the work. On quality, the C runtime's perplexity differs from the PyTorch reference by only 0.082%, with 99.41% greedy token match. The entire pipeline from data cleaning to C runtime is open source and reproducible, and the author openly admits the output is weak, currently supports only Windows x64, and invites others to reproduce or falsify with JSON results.