SlimWise Expert Pruning: Boosting MoE Decode Throughput by 1.8x

Mixture-of-Experts (MoE) models are supposed to gain inference speed from sparse activation, but in real serving scenarios during batch decoding, the scheduling overhead of routing expert traffic often cancels out the sparsity gains, making the latency bottleneck hard to break. SlimWise proposes a more refined approach: prune experts only during the decode phase, and hand off the complete KV cache generated during prefill directly to the pruned decoder—thereby nearly doubling decode throughput without sacrificing generation quality.

Why Full-Pipeline Pruning Fails

Previous serving optimizations either pruned the full model—including the costly prefill phase—or accepted a "quality-for-speed" trade-off. The problem is that the prefill phase plays a critical role in building rich context. If experts are disabled at the same ratio during this phase, the quality of the attention context the model accumulates drops noticeably.

In practice, this approach leads to higher perplexity or shorter output lengths, producing perceptible losses in generation fidelity. In other words, pruning itself isn't the problem—pruning the wrong phase is. The full computation during prefill is the foundation for high-quality decode-phase output, and once it's cut, no amount of downstream speedup can compensate for the semantic damage.

Understanding the trade-off here requires first clarifying the fundamental difference between the prefill and decode phases. The prefill phase processes the user's entire prompt, with all tokens computed in parallel over time. The attention layers build the complete KV cache in one pass—the key (Key) and value (Value) vectors for each token—for reuse across all subsequent generation steps. This phase determines how deeply the model understands the context; it is the "foundation" for downstream generation quality.

The decode phase is autoregressive: each generated token requires reading and appending to the KV cache. Computation per step is small, but there are many steps, making it highly sensitive to per-step latency. This is precisely why the two phases have entirely different bottlenecks—prefill is compute-bound, decode is memory-access and scheduling-bound.

The KV cache is the hinge connecting the two phases: it is both the product of prefill and the input to decode. Any pruning scheme that tries to reduce computation during prefill will contaminate this shared cache, forcing the decoder to start from inferior context. This is the technical root of why "pruning the wrong phase" is so costly.

SlimWise's Core Mechanism

SlimWise's approach can be summarized as "phase decoupling": the prefill phase remains untouched, letting the full model build a high-quality KV cache; the decode phase activates only half the experts for generation. Crucially, the decoder directly receives the full-model KV cache produced during prefill, inheriting a high-quality attention context.

According to the paper, on Qwen3.6-35B-A3B with a 50% expert pruning ratio, SlimWise boosts decode throughput by up to 1.81x with minimal accuracy loss. This token-level latency reduction translates to roughly 78% end-to-end speedup under typical request patterns.

The research also found that thanks to the KV cache handoff, the accuracy lost to pruning in many evaluation scenarios can be recovered without additional training—the pruned model performs on par with the unpruned baseline.

To understand why SlimWise can achieve "zero-training accuracy recovery," it helps to understand how experts work in MoE. Each MoE layer consists of multiple feed-forward sub-networks. A router scores each token and activates only the top-scoring experts—this is sparse activation. The "50% expert pruning ratio" in the paper refers to halving the number of activatable experts per layer during the decode phase, not deleting expert weights themselves—the pruned weights remain, they simply no longer participate in computation.

Notably, MoE's routing redundancy is typically high: different experts are functionally substitutable to a considerable degree, and for many tokens, rerouting to a suboptimal expert barely changes the output distribution. This is the precondition for maintaining quality at 50% pruning, and it explains why pruning often needs no retraining to stay close to baseline. The KV cache handoff works because the attention layers are completely untouched—the decoder receives context representations bit-for-bit identical to the full model, with quality loss stemming only from feed-forward path sparsification.

The Engineering Elegance of Zero Conversion Overhead

SlimWise has an easily overlooked but critical engineering detail: because expert pruning preserves the attention architecture and KV cache format, this KV cache handoff requires no cache conversion whatsoever.

This means the runtime only needs to switch weight masks between the two phases, eliminating expensive data reshaping and extra memory copies. In many pruning schemes, it is precisely these invisible data movement costs that devour the theoretical latency gains. By keeping the cache layout unchanged, SlimWise reduces these overheads to zero, turning pruning into genuinely observable throughput improvements.

Applicability Boundaries and Open Questions

SlimWise's validation covers two MoE backbones and three pruning criteria, but larger models or more aggressive pruning ratios remain untested. Additionally, its low-cost distillation stage, while helpful, adds an extra training step before deployment.

One open question worth watching: can dynamic token-level expert selection further tighten the latency-quality trade-off curve without relying on static pruning thresholds? This may be the next meaningful direction to explore.

The distillation stage mentioned in the paper refers to using the output distribution of the full model (teacher) to guide fine-tuning of the pruned model (student), so it can approximate the original predictions even after expert reduction. Even though the paper emphasizes that most scenarios recover accuracy without additional training, distillation remains as an optional "insurance policy"—at the cost of an extra training pipeline before deployment, requiring additional compute and data, and lengthening the model iteration cycle.

As for dynamic token-level expert selection, the idea is to let each token adaptively decide how many experts to activate at inference time based on its own difficulty: easy tokens compute less, hard tokens compute more. Compared to static thresholds, this approach could theoretically trace a better latency-quality curve, but the implementation challenges lie in the scheduling overhead introduced by routing decisions themselves, and the load imbalance from varying expert counts across tokens in a batch—the same class of challenge as the "scheduling overhead canceling sparsity gains" mentioned at the beginning of this article.

Implications for Production Deployment

For production systems currently serving MoE models, SlimWise sends a clear signal: replace the single decode pipeline with a prefill-decode disaggregated architecture, feeding the full-model KV cache directly into the pruned decoder, thereby eliminating runtime cache rewrite costs.

Re-running latency benchmarks under this decoupled paradigm will likely reveal additional speed headroom currently masked by end-to-end metrics. For inference services pursuing high throughput and low latency, the gains from phase decoupling may be more worth prioritizing than pure model compression.

One-line Summary

SlimWise prunes experts only during decode and hands off the full KV cache, nearly doubling decode throughput.

Paragraph Summary

SlimWise addresses the pain point in MoE model serving where sparse activation gains are canceled by scheduling overhead, proposing a phase-decoupled pruning strategy: keep full computation during prefill to build a high-quality KV cache, halve expert activation only during decode, and hand off the full-model KV cache from prefill directly to the pruned decoder. On Qwen3.6-35B-A3B, 50% expert pruning yields up to 1.81x decode throughput improvement, roughly 78% end-to-end speedup under typical request patterns, with minimal accuracy loss—recoverable without additional training in most scenarios. Engineering-wise, because the attention architecture and cache format remain unchanged, the KV cache handoff requires no conversion; the runtime merely switches weight masks. The article also notes that larger models and more aggressive pruning ratios remain to be validated, and recommends that production systems adopt a prefill-decode disaggregated architecture.