19.8MB LLM: How a 44M-Parameter Model Achieves 1900 tok/s on CPU

A complete model weighing just 19.8MB, running at roughly 1900 tok/s on a laptop CPU with only 41MB of memory — this isn't a compressed experimental artifact, but a fully trained language model that has processed 45 billion tokens from scratch. Reddit developer QLNI recently open-sourced SHADOW-50M (44M parameters in practice), attempting to answer an unconventional question: when you strip computation and persistent memory out of model weights and handle them separately, how much useful behavior can an extremely small local model actually deliver?

This isn't a benchmark competition. The author repeatedly emphasizes in the post: "I'm not saying a 20MB model can beat a normal LLM — it can't." But precisely because it abandons the race for general-purpose capability, the engineering philosophy behind SHADOW is worth unpacking.

Ternary Weights and Fingerprint Vocabulary: Two Pillars of Extreme Compression

SHADOW-50M achieves its 19.8MB footprint through two key design choices.

The first is ternary weights, where all parameters are constrained to {-1, 0, +1}. This aligns with the 1-bit/1.58-bit LLM ideas that have gained academic attention in recent years — trading extremely low precision for storage and inference speed. The author also ran a controlled experiment: a Llama-style model called Supra-50M with 51.8M parameters comes in at 103.6MB in bf16 and 56.2MB after 8-bit quantization, but dropping to int4 pushes perplexity from 165 to 193, and going all the way to ternary causes complete collapse. This demonstrates that ternary weights can't simply be obtained by post-hoc quantization — they require design decisions made from the training stage.

The second pillar is more counterintuitive: SHADOW doesn't use a learned embedding layer. Instead, it represents its vocabulary of 73,880 tokens using fixed 512-bit "fingerprints," with the entire frozen table occupying just 4.7MB. By comparison, Supra's vocabulary has only 32,000 tokens. The frozen fingerprint table offers a property that trained embeddings simply can't match — new tokens can be added at any time without retraining.

Technical note on ternary weights: The theoretical foundation comes from the BitNet line of work. Traditional neural networks store parameters as float32 or bfloat16, with each weight occupying 2–4 bytes. Ternary quantization constrains all weights to {-1, 0, +1}, theoretically requiring only 1.58 bits per parameter (log₂3). This means matrix multiplication degrades to addition, subtraction, and zero-skipping operations — no floating-point multipliers needed — yielding significant speedups on both specialized hardware and CPU SIMD instructions. The key challenge is training: directly quantizing a trained float32 model to ternary precision causes severe accuracy loss. This is why "training from scratch in ternary" is necessary — using ternary approximations in the forward pass and a Straight-Through Estimator in the backward pass to maintain gradient flow, letting the model learn representations under ternary constraints from the very beginning. This explains why Supra-50M's perplexity degrades sharply under post-hoc quantization, while SHADOW maintains usable performance.

An Unexpected Benefit of the Frozen Vocabulary: Patching 8,600 Missing Tokens Directly

The first version of SHADOW-50M was missing roughly 8,600 English word pieces — for instance, the lowercase "fitzgerald" would be incorrectly split into "fitz" before being fed into the model. The author tried fine-tuning to fix this, but every time the model learned a new token, it broke existing capabilities — a classic manifestation of catastrophic forgetting in neural networks.

The solution was clean and elegant: simply append 8,600 rows to the frozen fingerprint table, with no training and no weight changes. All 34 previously published answers remained identical, while the model could now correctly process a large number of new token pieces. Even more interesting: the fingerprint table scored a Spearman correlation of 0.594 on human word similarity benchmarks, compared to -0.057 for random encoding — indicating that these fixed fingerprints carry meaningful semantic structure rather than serving as random hashes.

"A trained embedding cannot accept thousands of new rows without training, but a frozen table can." This line from the author captures the entire design philosophy: liberating extensibility from the parameters.

Computation and Memory: Delegated to Dedicated Circuits, Not Weights

SHADOW's most central innovation is its refusal to use neural network weights for memorizing arithmetic or storing facts.

When the model determines that a computation is needed, it outputs a token like [calc]347*86[eq], and a fixed circuit takes over during the readout phase to fill in the correct number within the same token stream. No calculator API, no tool calls, no pasting results back into the prompt. The author built circuits for arithmetic, percentages, dates, days of the week, unit conversion, counting, sorting, comparison, and even a small programmable machine.

The memory system similarly bypasses mainstream approaches. When storing a record, the model reads it once and writes the attention states to disk at 1-bit precision — 288 bytes per token. Retrieval is triggered by a [need] marker that hits an index in roughly 1 microsecond; the stored attention states are injected directly back into the model in about 0.03ms, without re-reading the text. The index costs just 22 bytes per token, with no vector database and no embedding model needed. At the 100 million token scale, the archive takes 28.8GB plus a 2.2GB index on disk, while the process memory through memory-mapping stays around 28MB — because a single query only touches the pages it needs.

The index also maintains a persistent trace: records that get accessed are reinforced in the index. On repeated queries, the top-1 hit rate climbs from 0.571 to 0.743 — with no model training whatsoever.

Technical note on disk-stored attention states: Writing attention states rather than text directly to disk is the core innovation of SHADOW's memory approach, and it differs fundamentally from mainstream RAG (Retrieval-Augmented Generation) architectures. Traditional RAG must re-feed retrieved text into the model during inference, requiring a full forward pass to reconstruct context representations — an expensive operation. SHADOW's approach serializes the intermediate attention activations produced after the model has processed the text during the "write" phase, then skips token re-reading at query time and injects those activation states directly into the corresponding network layers — essentially caching and reusing the model's internal state of "having already read this passage." Quantizing attention states to 1-bit precision (288 bytes per token) is a tradeoff between storage efficiency and fidelity, allowing a 100M-token archive to occupy roughly 28.8GB with on-demand loading via mmap, keeping process memory extremely low. This approach shares conceptual similarities with an extended KV-Cache, but persisted to disk to form permanent cross-session memory.

Head-to-Head Results: Lost on Benchmarks, Won on Real Tasks

The author honestly documented the shortcomings. On standard evaluations, Supra-50M leads across the board: ARC-Easy 0.435 vs. SHADOW's 0.307, PIQA 0.600 vs. 0.570, WikiText-2 perplexity 165 vs. 186.

But the comparisons that actually reflect the design intent are the practical tasks. Asked "I had 3 books and bought 5 more, how many do I have?", SHADOW answers "8" directly, while Supra responds with "Books usually come from collections of short stories, poetry, or other literary forms." Asked "What is 15% of a $240 bill?", SHADOW answers "$36," while Supra produces a self-contradictory "$250... $150." When queried about stored patient record "P-204," SHADOW accurately retrieves "asthma" from disk, while Supra — with the record pasted directly into the prompt — hallucinates "the patient has asthma when sleeping too much." For a nonexistent patient Z-999, SHADOW clearly responds "no record found," while Supra fabricates something about the immune system.

This set of comparisons defines SHADOW's positioning: it's weak on open-domain knowledge and creative writing, but in scenarios requiring precise computation and reliable retrieval, the combination of dedicated circuits and persistent memory is far more dependable than asking a small model to memorize everything.

Technical note on the benchmark paradox: This comparison reveals a deep tension in language model evaluation. Standard benchmarks like ARC and PIQA measure world knowledge and linguistic reasoning internalized from pretraining data — a category where a 44M-parameter model with limited training scale is almost inherently at a disadvantage. SHADOW deliberately sacrifices exactly this capability, instead achieving reliability on "tasks requiring precise answers" through external deterministic mechanisms. This reflects a philosophical divergence in engineering: general-purpose LLMs try to cover all task types through statistical learning, including computation and factual recall; SHADOW takes the position that for tasks with definite answers, deterministic algorithms are naturally superior to probabilistic inference and shouldn't waste neural network capacity learning multiplication tables or memorizing specific facts. Supra-50M's hallucination on the memory task — getting the correct information wrong even when it's provided directly in the prompt — further illustrates that at extremely small parameter scales, forcing a model to handle tasks beyond its capability boundary leads to unpredictable failure modes.

Four Application Experiments: Small Models Can Work Alongside Large Ones

The author built four test setups pairing this 20MB model with larger models.

The most interesting is video memory: Gemma 3 4B watches a ten-minute short film, samples one frame every two seconds, and writes 298 descriptions like "Moment M-0039: a chubby white rabbit reaches for a purple butterfly" — then Gemma exits. SHADOW retains these 298 moments as disk memory and can subsequently answer questions by ID, timestamp, or scene content, without the video or Gemma, running on a laptop with roughly 42MB of memory and scoring 55–56 out of 60.

The other three experiments: an inventory query across 1,600 records achieved 159/160, with all 20 questions about never-stored items correctly returning "no record"; SHADOW as a draft model for Qwen3-32B boosted llama.cpp generation speed from 19.7 to 28.5 tok/s while Qwen still decided every final token; and an MCP memory server let Qwen3-14B store facts mid-conversation and later retrieve them with a 5/5 hit rate while citing original records.

The same kernel compiled to WebAssembly runs in a browser tab at roughly 500 tok/s, fully offline. Even more surprisingly, a contributor named engram-forge submitted a PR containing a CUDA engine, quantization tutorials, and — a talking Peppa Pig plush toy with SHADOW stuffed inside, built on a ~$35 board with a microphone and speaker, generating responses locally with no cloud, no account, and no internet. The author hasn't merged it yet, as they can't independently verify the ~11,000 lines of CUDA code.

Technical note on speculative decoding: SHADOW's role as a draft model to accelerate Qwen3-32B uses speculative decoding. The principle: a small "draft model" rapidly generates several candidate tokens, which the large "target model" then verifies in parallel in a single pass. Since verification is far cheaper than autoregressive generation, overall throughput improves as long as the draft model's acceptance rate is high enough. The theoretical speedup depends on acceptance rate: if the draft correctly predicts k consecutive tokens, the large model effectively gets k steps "for free." SHADOW's success as a draft model shows that even though its language capability falls short of Qwen3-32B, the two models have sufficient overlap in token distributions to achieve meaningful acceleration (19.7→28.5 tok/s, roughly a 45% improvement). This also demonstrates that extremely small models have real value in the specific role of large-model inference acceleration, not just as standalone assistants.

Limitations and Open Source

The author documented the weaknesses in the repository as well: general knowledge is thin, creative writing is poor, multi-digit arithmetic occasionally copies digits incorrectly, and large archives sometimes pull in irrelevant records for questions containing numbers.

The entire project is MIT-licensed, with primary weights and fine-tuning/export tooling already public and a browser demo available for direct testing. The author says training code, datasets, the frozen table, and a complete technical writeup — including costs, failed experiments, and mistakes — are forthcoming.

SHADOW-50M's value isn't in benchmark scores. It validates a path that the mainstream has largely ignored: rather than making ever-larger models carry everything themselves, hand computation off to deterministic circuits and memory off to disk indexes, and let the neural network focus on what it actually excels at — language understanding. For on-device AI and offline scenarios, this "division of labor" philosophy may have more headroom than blindly scaling up parameters.