Expert Pruning
A technique for reducing the activatable expert capacity in mixture-of-experts models by decreasing the number of experts involved in computation to lower inference overhead; the pruned expert weights are retained but do not participate in computation.
Timeline (last 90 days)
Most existing expert pruning methods aggregate importance by token and prune by global ranking; this uniform aggregation lets the modality with more tokens or stronger attention dominate pruning decisions, potentially harming useful experts in the other modality.
Expert pruning has a trade-off curve: pruning less means smaller performance loss but limited memory savings, while pruning more means noticeable memory reduction but possibly hitting a capability floor; different layers have inconsistent sensitivity to pruning, and degradation may collapse suddenly under extreme compression.
In expert pruning, whether an expert is important depends on the routing distribution and can be evaluated based on activation frequency, routing probability mass, or output contribution on a calibration set.
In the post, the author actively asks the community for better expert selection methods, and the optimal expert selection strategy has not yet been determined.
Under a large sampling budget, different expert pruning strategies may cover the same set of problems, differing only in the number of attempts required.
The study systematically compared expert pruning, FFN pruning, and layer pruning on four categories of math and coding benchmarks.
Expert pruning typically targets mixture-of-experts (MoE) models, removing redundant or unimportant experts to compress model size.
Traditional expert pruning methods score and rank each expert independently or exhaustively search expert combinations, ignoring the global structural relations among experts.
Exploiting structural redundancy can effectively compress the number of experts and reduce memory usage without significant performance loss.
This work reframes expert pruning as a structure-preserving expert subset selection problem and optimizes it efficiently with a greedy strategy.
All Facts (10)
Most existing expert pruning methods aggregate importance by token and prune by global ranking; this uniform aggregation lets the modality with more tokens or stronger attention dominate pruning decisions, potentially harming useful experts in the other modality.
50%UnverifiedExpert pruning has a trade-off curve: pruning less means smaller performance loss but limited memory savings, while pruning more means noticeable memory reduction but possibly hitting a capability floor; different layers have inconsistent sensitivity to pruning, and degradation may collapse suddenly under extreme compression.
50%UnverifiedIn expert pruning, whether an expert is important depends on the routing distribution and can be evaluated based on activation frequency, routing probability mass, or output contribution on a calibration set.
50%UnverifiedIn the post, the author actively asks the community for better expert selection methods, and the optimal expert selection strategy has not yet been determined.
50%UnverifiedUnder a large sampling budget, different expert pruning strategies may cover the same set of problems, differing only in the number of attempts required.
50%UnverifiedThe study systematically compared expert pruning, FFN pruning, and layer pruning on four categories of math and coding benchmarks.
50%UnverifiedExpert pruning typically targets mixture-of-experts (MoE) models, removing redundant or unimportant experts to compress model size.
50%UnverifiedTraditional expert pruning methods score and rank each expert independently or exhaustively search expert combinations, ignoring the global structural relations among experts.
50%UnverifiedExploiting structural redundancy can effectively compress the number of experts and reduce memory usage without significant performance loss.
50%UnverifiedThis work reframes expert pruning as a structure-preserving expert subset selection problem and optimizes it efficiently with a greedy strategy.
50%