[KongchangAI]
ConceptRAMP-based Pruning

RAMPr

A RAMP-based neuron-level pruning technique for MoE models that selects domain-specialized neurons to balance the retention of capabilities across multiple domains.

Timeline (last 90 days)

Nov 11

The researchers propose a RAMP-based pruning technique, RAMPr, to select neurons across experts to balance the preservation of multi-domain capabilities.

Unverified50%
Nov 11

At a 75% neuron retention rate, RAMPr retains on average 91-98% of the original full performance across six models.

Unverified50%
Nov 11

When the neuron retention rate drops to 50%, RAMPr can reduce resident GPU memory by 36-43%.

Unverified50%
Nov 11

The advantage of RAMPr lies in pruning directly targeting domain-specialized neurons, avoiding excessive damage to multi-domain capabilities.

Unverified50%
Nov 11

The RAMPr code is open-sourced on GitHub at the repository ydyjya/Neuron-Level-Specialization-in-MoE.

Unverified50%

All Facts (5)

Source Articles