RAMPr
A RAMP-based neuron-level pruning technique for MoE models that selects domain-specialized neurons to balance the retention of capabilities across multiple domains.
Timeline (last 90 days)
The researchers propose a RAMP-based pruning technique, RAMPr, to select neurons across experts to balance the preservation of multi-domain capabilities.
At a 75% neuron retention rate, RAMPr retains on average 91-98% of the original full performance across six models.
When the neuron retention rate drops to 50%, RAMPr can reduce resident GPU memory by 36-43%.
The advantage of RAMPr lies in pruning directly targeting domain-specialized neurons, avoiding excessive damage to multi-domain capabilities.
The RAMPr code is open-sourced on GitHub at the repository ydyjya/Neuron-Level-Specialization-in-MoE.
All Facts (5)
The researchers propose a RAMP-based pruning technique, RAMPr, to select neurons across experts to balance the preservation of multi-domain capabilities.
50%UnverifiedAt a 75% neuron retention rate, RAMPr retains on average 91-98% of the original full performance across six models.
50%UnverifiedWhen the neuron retention rate drops to 50%, RAMPr can reduce resident GPU memory by 36-43%.
50%UnverifiedThe advantage of RAMPr lies in pruning directly targeting domain-specialized neurons, avoiding excessive damage to multi-domain capabilities.
50%UnverifiedThe RAMPr code is open-sourced on GitHub at the repository ydyjya/Neuron-Level-Specialization-in-MoE.
50%