PEEP
An MoE expert pruning method that preserves functional subspaces by maximizing weighted projection energy, using router weights as a prior to select a structure-preserving subset of experts.
Timeline (last 90 days)
Projection Energy Expert Pruning (PEEP) preserves functional subspaces by maximizing weighted projection energy.
Both PEEP and DCEP introduce router weights as priors to balance the global structural relations among experts and the local importance of individual experts.
At expert retention ratios of 75% and 50%, PEEP and DCEP surpass existing state-of-the-art baselines on most benchmarks and achieve the best average performance on general tasks and mathematical reasoning tasks.
The greedy optimization strategies provided by PEEP and DCEP are computationally efficient and suitable for practical deployment.
All Facts (4)
Projection Energy Expert Pruning (PEEP) preserves functional subspaces by maximizing weighted projection energy.
50%UnverifiedBoth PEEP and DCEP introduce router weights as priors to balance the global structural relations among experts and the local importance of individual experts.
50%UnverifiedAt expert retention ratios of 75% and 50%, PEEP and DCEP surpass existing state-of-the-art baselines on most benchmarks and achieve the best average performance on general tasks and mathematical reasoning tasks.
50%UnverifiedThe greedy optimization strategies provided by PEEP and DCEP are computationally efficient and suitable for practical deployment.
50%