A New Perspective on Expert Pruning: Structural Redundancy and Preservation Framework for Large MoE Models

Sparse Mixture-of-Experts (MoE) architectures scale model capacity while improving inference efficiency, but deployment still requires storing all experts, causing significant memory overhead. A recent study points out that expert outputs in MoE layers exhibit structural redundancy—across different models and task categories, the stable rank and entropy effective rank of expert outputs are consistently far lower than the nominal number of experts, and the effective modes of the output distribution are also fewer. This finding provides a new theoretical basis for expert pruning.

From Independent Scoring to Structured Subset Selection

Traditional expert pruning methods typically score and rank each expert independently, or exhaustively search expert combinations, ignoring the global structural relationships among experts. This study reframes expert pruning as a structure-preserving expert subset selection problem and uses a greedy strategy for efficient optimization. This framework no longer treats individual experts in isolation but views experts as nodes in an overall structure, preserving key structural information during pruning.

The researchers propose two pruning methods under a unified framework:

• Projection Energy Expert Pruning (PEEP): Preserves functional subspaces by maximizing weighted projection energy, ensuring that the pruned expert set still covers the main directions of the original functional space.
• Distribution Coverage Expert Pruning (DCEP): Preserves distributional geometry by minimizing weighted coverage error, making the output distribution of the pruned experts approximate the original distribution as closely as possible.

Both methods introduce router weights as a prior, thereby preserving the global structural relationships among experts while accounting for the local importance of individual experts.

Stable rank is the ratio of the squared Frobenius norm to the squared spectral norm of a matrix, i.e., (|W|_F^2 / |W|_2^2). It does not exceed the nominal rank of the matrix and is sensitive to singular value decay. Unlike hard rank, it reflects the degree of energy concentration: if a few singular values dominate, the stable rank is far smaller than the matrix dimension. In this article, it is used to continuously quantify the low-rankness of expert output matrices.

Entropy effective rank is based on the Shannon entropy after normalizing singular values: the squared singular values are normalized into a probability distribution, its entropy is computed, and the exponential is taken to obtain the effective number of dimensions. The more uniform the distribution, the closer the effective rank is to the nominal dimension; the more concentrated it is, the smaller the effective rank. It is complementary to stable rank, and together they characterize the degree of structural redundancy in expert outputs.

Weighted projection energy (PEEP) is based on the idea of treating the outputs of retained experts as a projection onto the original functional subspace, weighting each expert's projection contribution by router weights so that the selected subset maximizes projection energy along the main functional directions. Weighted coverage error (DCEP) measures the difference between the weighted output distribution of the selected experts and the full-expert distribution; by minimizing this error, the subset covers the original distribution as much as possible in distributional geometry. Both use router weights as a prior, unifying global structural constraints and local importance into the same greedy selection criterion.

Experimental Validation: Performance Across Models and Tasks

The study was validated on two models: DeepSeek^Ad-V2-Lite and Qwen3-30B-A3B. With expert retention ratios of 75% and 50%, PEEP and DCEP outperformed existing state-of-the-art baselines on most benchmarks and achieved the best average performance on both general tasks and mathematical reasoning tasks. This shows that exploiting structural redundancy can effectively compress the number of experts and reduce memory usage without significantly sacrificing performance.

The core contribution of this work lies in revealing the low-rank property of MoE expert outputs and elevating pruning from "individual importance evaluation" to "structure-preserving subset selection," offering a new approach for efficient deployment of large MoE models.

DeepSeek-V2-Lite is a lightweight MoE model in the DeepSeek-V2 series, with about 16B total parameters and about 2.4B activated parameters. It adopts MLA attention and a fine-grained expert design, making it a representative base for MoE pruning research. Qwen3-30B-A3B is an MoE model in the Qwen3 series, with about 30B total parameters and about 3B parameters activated per token, featuring finer expert granularity and sparser routing. The two differ in scale, number of experts, and routing mechanism, allowing the generalization of pruning methods to be tested across different MoE configurations.

Implications for MoE Deployment

For enterprises and developers deploying large MoE models, the discovery of structural redundancy means that by retaining key structural subsets, most of the model's capabilities can be maintained in memory-constrained environments. The greedy optimization strategies provided by PEEP and DCEP are computationally efficient and suitable for practical applications. In the future, how to extend the structure-preserving framework to more model architectures and dynamic routing scenarios remains a direction worth exploring.

One-line Summary

The study finds structural redundancy in MoE expert outputs and proposes two structure-preserving pruning methods, PEEP and DCEP.

Paragraph Summary

The article introduces a study on expert pruning for sparse Mixture-of-Experts (MoE) models. The study finds structural redundancy in expert outputs within MoE layers: stable rank and entropy effective rank are far lower than the nominal number of experts, and the effective modes of the output distribution are fewer. Based on this, the researchers reframe expert pruning as a structure-preserving expert subset selection problem and optimize it with a greedy strategy, proposing Projection Energy Expert Pruning (PEEP) and Distribution Coverage Expert Pruning (DCEP), both of which introduce router weights as a prior. On DeepSeek-V2-Lite and Qwen3-30B-A3B, at 75% and 50% retention ratios, they outperform existing state-of-the-art baselines on most benchmarks and achieve the best average performance on general tasks and mathematical reasoning tasks, offering a new approach for memory-constrained deployment of large MoE models.