SlimWise
An expert pruning optimization method for Mixture-of-Experts (MoE) model inference serving. It decouples the prefill and decode phases, reuses KV cache, and applies lightweight distillation to improve decode throughput without requiring full model retraining.
Timeline (last 90 days)
Although SlimWise's low-cost distillation stage is helpful, it adds an extra training step before deployment, requiring additional compute and data.
Thanks to the KV cache handoff, in many evaluation scenarios the accuracy lost due to pruning is recovered without additional training, and the pruned model performs comparably to the unpruned baseline.
SlimWise's validation covers two MoE backbones and three pruning criteria, but larger models or more aggressive pruning ratios have not yet been tested.
SlimWise prunes experts only during the decoding stage, and hands the complete KV cache generated during the prefill stage directly to the pruned decoder.
When run on Qwen3.6-35B-A3B with a 50% expert pruning ratio, SlimWise increases decoding throughput by up to 1.81x with minimal accuracy loss.
Under this pruning scheme, the token-level latency reduction translates into roughly 78% end-to-end speedup under typical request patterns.
Because expert pruning preserves the attention architecture and KV cache format, the KV cache handoff requires no cache conversion, and the runtime only needs to switch weight masks between the two stages.
The "50% expert pruning ratio" in the paper means halving the activatable expert capacity of each layer during the decoding stage, rather than deleting the expert weights themselves.
Previous claims about SlimWise achieving approximately 78% latency reduction have been identified as inaccurate; the actual benefit should be measured by 1.81× decoding throughput.
SlimWise prunes experts during the inference serving stage and decouples the prefill and decode phases, combined with KV cache reuse and lightweight distillation, achieving up to 1.81× decoding speedup with minimal accuracy loss.
All Facts (10)
Although SlimWise's low-cost distillation stage is helpful, it adds an extra training step before deployment, requiring additional compute and data.
50%UnverifiedThanks to the KV cache handoff, in many evaluation scenarios the accuracy lost due to pruning is recovered without additional training, and the pruned model performs comparably to the unpruned baseline.
50%UnverifiedSlimWise's validation covers two MoE backbones and three pruning criteria, but larger models or more aggressive pruning ratios have not yet been tested.
50%UnverifiedSlimWise prunes experts only during the decoding stage, and hands the complete KV cache generated during the prefill stage directly to the pruned decoder.
50%UnverifiedWhen run on Qwen3.6-35B-A3B with a 50% expert pruning ratio, SlimWise increases decoding throughput by up to 1.81x with minimal accuracy loss.
50%UnverifiedUnder this pruning scheme, the token-level latency reduction translates into roughly 78% end-to-end speedup under typical request patterns.
50%UnverifiedBecause expert pruning preserves the attention architecture and KV cache format, the KV cache handoff requires no cache conversion, and the runtime only needs to switch weight masks between the two stages.
50%UnverifiedThe "50% expert pruning ratio" in the paper means halving the activatable expert capacity of each layer during the decoding stage, rather than deleting the expert weights themselves.
50%UnverifiedPrevious claims about SlimWise achieving approximately 78% latency reduction have been identified as inaccurate; the actual benefit should be measured by 1.81× decoding throughput.
50%UnverifiedSlimWise prunes experts during the inference serving stage and decouples the prefill and decode phases, combined with KV cache reuse and lightweight distillation, achieving up to 1.81× decoding speedup with minimal accuracy loss.
50%