Training a CPU-Native LLM on Zero Budget: The Architectural Gamble of Ternary Quantization × Sparse MoE

A Counterintuitive Bet

At a time when GPU compute is treated as the hard currency of large model training, one independent developer is publicly running what looks like an impossible experiment on Reddit: training a CPU-native large language model on a $0 budget — and trying to prove it can scale.

This is not just another "small-model toy" project. The developer's core thesis is precise: through co-designed architecture, the decoding speed per token should depend only on the "active parameters" rather than the total parameter count. If this bet pays off, it could open an entirely new path for LLM deployment in compute-constrained environments.

Three-Layer Technical Stack: The Architecture at Its Core

Keeping Active Slices Resident in L3 Cache

The design philosophy of this model rests on the combination of three techniques: ternary quantization + sparsity + fine-grained Mixture of Experts (MoE).

Ternary quantization compresses neural network weights to only three discrete values: {-1, 0, +1}. Compared to 32-bit FP32 floats, this reduces each weight's storage from 32 bits to roughly 1.58 bits (the theoretical limit), cutting memory footprint by about 20×. Recent work like BitNet b1.58 has brought this technique to large language models. Its key advantage: weight multiplication degrades to addition, subtraction, and conditional skips (multiplying by 0), which on a CPU means bypassing expensive floating-point multiply units and dramatically improving computational efficiency.

The most critical insight is this — the entire architecture is carefully "trimmed" so that the active slice of parameters involved in each inference step fits precisely into the CPU's L3 cache. To understand why this matters, consider the CPU memory hierarchy from fastest to slowest: registers → L1 cache (~KB, ~1ns latency) → L2 cache (~MB, ~5ns latency) → L3 cache (~MB to tens of MB, ~30ns latency) → main memory DRAM (~GB, ~100ns latency). The primary bottleneck in neural network inference is typically not arithmetic throughput but memory bandwidth. A modern desktop CPU's L3 cache bandwidth can reach hundreds of GB/s, while DDR5 main memory bandwidth typically sits at 50–100 GB/s — a gap of 5–10×. Once the active parameters can remain resident in fast cache, the massive latency of main memory access is bypassed, enabling meaningful throughput on an ordinary desktop CPU.

The developer currently has one empirically measured data point: an 8.3M-parameter sandbox engine achieving approximately 2,000 tokens/second decoding speed on a 6-core desktop CPU from 2019. For pure CPU inference, that's a genuinely impressive number.

Fine-Grained MoE: More Capacity, Same Speed

The core bet of the entire experiment hinges on the scaling properties of fine-grained MoE. The central idea behind Mixture of Experts (MoE) architecture is to replace feed-forward layers with multiple parallel "expert" sub-networks, using a lightweight routing network to dynamically select a small subset of experts for each input token while the rest remain silent. This way, total parameter count can scale substantially while the number of parameters actually activated per inference step stays constant. Models like Mixtral 8×7B and DeepSeek-MoE have already validated this paradigm. "Fine-grained MoE" takes this further by shrinking individual expert size and increasing their count, making routing selection more flexible and precise — the key mechanism for compressing the active slice down to L3-cache-resident size.

The developer's hypothesis: as total parameter count grows from 105M to 206M to 402M, as long as the active slice size remains fixed, decoding speed should remain constant and stay reliably above a "usable baseline."

In other words, the model's "knowledge capacity" can keep expanding while the "real-time cost" of running it stays nearly flat. This is the most compelling promise of MoE architecture — but whether it holds under such extreme constraints on a CPU remains an open question. The developer openly acknowledges that beyond the 8.3M measured data point, all other numbers are "desk predictions" based on bandwidth modeling — calibrated against the measured point but not yet validated by actual training.

Extreme Resource Constraints

Perhaps the most striking aspect of this project isn't the technology itself, but the resource environment it operates in. As the developer puts it, the constraints are "almost comically tight":

• 2 free Kaggle T4 GPUs
• A friend's RTX 5090 GPU, available only for limited use (he specifically thanks this friend, Ale)
• Total budget: $0

The training approach uses offline distillation — pre-generating a large labeled dataset from a teacher model and then training the student on it — avoiding the compute demands of online distillation, which requires synchronous teacher model inference. The teacher model is Qwen2.5-Coder under the Apache 2.0 license. Currently, the teacher model's scoring phase runs on an entry-level RTX 3060.

Under these hardware constraints, attempting to train a progressively scalable model ladder is in itself a dual test of engineering skill and architectural efficiency.

Pre-Registered Criteria: Scientific Discipline in Practice

One of the most commendable aspects of this project is the developer's commitment to research rigor.

He established a pre-registered success/failure criterion for the experiment: "Cross-tokenizer distillation must outperform plain cross-entropy loss." Cross-tokenizer distillation is a technically tricky problem: when the student and teacher models use different vocabularies, their output probability distributions can't be directly aligned — a mapping between token spaces must be constructed. This is precisely why the developer chose it as the criterion — if this approach doesn't work, the validity of the entire training scheme is in question.

This criterion was pushed to the Git repository before training began, specifically to prevent himself from quietly moving the goalposts after seeing results.

Pre-registration originated in psychology and medical clinical trials as an institutional safeguard against research biases like data dredging and HARKing (Hypothesizing After Results are Known). The "Replication Crisis" of the 2010s helped spread this practice across many academic disciplines. Bringing it into open-source AI development is a valuable attempt to transplant academic rigor into engineering practice — because regardless of outcome, conclusions are reached after the criteria are locked in, not reverse-engineered to fit a successful result.

The developer states explicitly: "If it fails, I'll publish that too, with full data." This attitude transforms the experiment from a technical showcase into a publicly falsifiable validation. Whatever happens, the community gets real, reproducible data.

The Deeper Significance of This Bet

From an industry perspective, this project touches on several directions worth watching:

First, the potential revival of CPU inference. As quantization techniques (especially ternary/binary) mature and MoE architectures enable fine-grained control over active parameters, CPUs may become a viable option for LLM deployment in specific scenarios — a development that carries significant implications for edge devices, privacy-sensitive applications, and GPU-free environments.

Second, the value of co-designed architectures. Rather than treating quantization, sparsity, and MoE as independent optimization techniques, the developer co-designed them around a unified goal: keeping the active slice in L3 cache. This approach — deriving model structure from hardware characteristics — represents an important paradigm in efficiency optimization.

Third, the demonstrative value of zero-budget innovation. Beyond the arms race of large model training with million-dollar compute budgets, this project is a reminder that algorithmic and architectural innovation can still generate genuine technical insight at extremely low cost.

Of course, as the developer repeatedly emphasizes, this bet could very well fail. Bandwidth model predictions may not hold; whether fine-grained MoE's routing behavior, sparsity, and per-parameter quality can be maintained at small scale and across scaling steps all remain open questions. But regardless of outcome, the practice of "making a public commitment, pre-registering criteria, and reporting results honestly" is itself the healthiest kind of open-source community practice.

This is worth watching closely.

Key Takeaways