After Pruning 85% of Gemma's Weights, the Model Got Faster Instead

A MoE model was cut down to just 15% of its parameters, yet still maintains 92% agreement with the original model on a classification task—a figure that even approaches the natural 90% difference between the 12B and 26B models. A developer on Reddit has published this aggressive pruning experiment, with the goal not of building a general-purpose model, but of finding a lighter, more power-efficient inference solution for an application running on-device 24/7.

What the Experiment Did: Turning MoE into a Dense Model

The pruning target is Gemma 4 26B-A4B, a Mixture-of-Experts (MoE) architecture model. The author's approach is straightforward: MoE activates only 8 experts per layer by default, so if you keep only 8 experts per layer and force all tokens through those 8 experts, the router loses its purpose, and the model degenerates into a dense network.

The result is 3.81B parameters, only 2.75 GB after Q4_K_M quantization. Inference speed on a Mac reaches 79 tokens/s, about twice that of the 12B model, while memory usage is only about half of the latter (3.8 GB).

The cost of this trade-off is equally clear: general-purpose tasks requiring knowledge reserves will degrade noticeably, because the knowledge carried by the deleted weights no longer exists. The author himself emphasizes that this is not a general-purpose model; it serves a specific application scenario—a large model running persistently on-device, where memory and energy consumption are hard constraints.

The MoE (Mixture-of-Experts) architecture centers on placing multiple sets of "expert" sub-networks in parallel within the Transformer's FFN layers, with a router selectively activating a small number of them for each token. The "A4B" in Gemma 4 26B-A4B means total parameters of about 26B, but only about 4B parameters activated per token on average—this is the origin of MoE's "many parameters, little computation." Since the router activates only 8 experts per layer by default, unselected experts do not participate at all in that forward pass. Therefore, directly pruning each layer to 8 experts and forcing all tokens to share them bypasses the routing decision and collapses the sparsely-activated MoE into a dense network: parameter count drops dramatically, and the computation path changes from "dynamically selected per token" to "fixed."

Q4_K_M, which appears repeatedly in the article, is a 4-bit quantization scheme in the llama.cpp ecosystem; K indicates k-quant block quantization, and M is the medium-quality tier. It compresses the original 16-bit weights to an average of about 4.5 bits, reducing memory/VRAM usage to about a quarter of the original with controllable precision loss. This is why the 3.81B-parameter model is only 2.75 GB after quantization—a key prerequisite for it to run persistently on consumer-grade devices; the 79 tokens/s speed and 3.8 GB memory usage cited in the article were also measured under this quantization format.

Why Block-Local Training Instead of Fine-Tuning

The author takes a negative view of the fine-tuning approach, and the reason lies in the design of the training objective. He uses block-local training: rather than training the model to output the correct answer, he trains it to reproduce the original "thinking process." In his words—require the model to perform the same reasoning, not to arrive at the same answer.

This distinction is especially critical for narrow tasks. If you only align the final labels, the model may learn to fit the data using completely different shortcuts; aligning block-level computation processes is equivalent to requiring each layer to produce the same output given the same input, preserving the original model's computation path.

All training data comes from the author's own dataset. He mentions that this method is cheap enough to be completed on users' own devices with their own data—requiring only about 5 GB of memory—with the aim of mitigating the distribution mismatch between deployment environment data and training data. However, he also admits that this has not yet been validated on real user devices.

Block-local training is a layer-by-layer, block-by-block supervision approach: instead of basing the loss on the model's final output labels, it targets the original model's output at each layer (or each Transformer block), requiring the pruned model to reproduce these intermediate activations given the same input. This shares the same lineage as the idea of "aligning logits or intermediate features" in knowledge distillation, but places more emphasis on locality—each time only one block needs to be trained while the rest are frozen, greatly reducing memory requirements (which also explains the claim that about 5 GB of memory suffices for training). Compared with end-to-end fine-tuning, it preserves the original model's layer-by-layer computation path and prevents the model from degenerating into shortcut label fitting on narrow tasks.

What 92% Agreement Means

Looking at the number 92% alone can easily lead to overestimating or underestimating the effect. The author provides a good frame of reference: on the same classification task, the agreement between the 12B and 26B models is 90%. In other words, the deviation between the pruned model and the original model is on the same order as the deviation between a smaller model in the same family and its bigger sibling.

This suggests an evaluation approach—whether pruning is "acceptable" should not be compared only with the original model, but with the alternatives you can accept. If your alternative is the 12B model, then a 2.75 GB model with double the speed achieving similar agreement becomes quite attractive.

It should be noted that this conclusion is strictly limited to a single classification task. Multi-task, open-domain question answering, and scenarios requiring factual recall are outside the scope of validation, and the author makes no such claims.

Expert Selection: An Open Problem

Regarding which 8 experts to keep per layer, the author's method is "pick a set, then use them all." At the end of the post, he proactively asks the community for help: are there better expert selection methods?

This question is indeed at the core of MoE pruning. Expert importance varies by task; selecting by activation frequency, routing weight, or contribution on calibration data may yield different results. The author's choice to fix 8 experts is partly to align with the default activation count, minimizing architectural changes. But for other tasks, this choice may not be optimal.

Another noteworthy detail is the transferability of the training method. Block-local training is theoretically model-agnostic; if personalized calibration can be completed on consumer-grade devices, the "fixed-at-factory" problem of on-device models would have a new solution. Of course, the claim of training with 5 GB of memory remains at the design level for now, and practical feasibility requires more validation.

Full technical details can be found in the author's blog post: Frankensteining Gemma's Experts.

Expert pruning is a specialized problem in MoE compression: unlike dense models where weights are pruned by magnitude, whether an expert in MoE is important depends on the routing distribution, and must be evaluated based on activation frequency, routing probability mass, or output contribution on a calibration set. Common approaches include ranking by average routing weight, counting activations on calibration data, or measuring information content by the variance of expert outputs. The author's approach of "fixing 8 experts and enabling them all" is a conservative choice aligned with the default activation count and requiring minimal changes, but it implicitly assumes all experts are equally important to the target task; re-ranking by task-relevant contribution might yield better agreement with only a slight increase in modifications.