Beyond Expert Boundaries in MoE Models: New Findings on Neuron-Level Specialization and Pruning

New research proposes RAMP to reveal neuron-level domain specialization in MoE models and develops RAMPr pruning.
Addressing the controversy over expert specialization in MoE models, this study proposes the RAMP method, shifting observation from the expert level to the neuron level. Domain-specialized neuron sets show 58-74% within-domain overlap versus only 15-24% across domains. Neuron importance reorganizes during pretraining but consolidates during post-training. The resulting RAMPr pruning maintains 91-98% performance at 75% retention and cuts GPU memory by 36-43% at 50%.
The nature of expert specialization in Mixture-of-Experts (MoE) models has long been debated: some reports claim that experts prefer domain-related inputs, while other analyses suggest that routing mainly depends on token identity or syntactic patterns. To clarify these disagreements, a new study shifts focus from expert-level routing to neuron-level organization, proposing the Routing-Aware Magnitude Profiling (RAMP) method and developing a pruning technique called RAMPr based on it.

Expert-Level Controversy and the Neuron-Level Perspective
Previous discussions about expert specialization in MoE models have mostly remained at the expert level: whether an expert prefers inputs from a certain domain (such as code, math, or multilingual text). But the study points out that this expert-level preference may be superficial—the unit that truly determines computational specialization lies at a finer granularity: the neuron.
The RAMP method quantifies each neuron's contribution to a specific domain by combining routing information with neuron activation magnitude. Using mixed-domain calibration data, researchers identified domain-specialized neuron sets: the overlap between samples from the same domain across different sources reached 58-74%, while cross-domain overlap was only 15-24%. This data indicates that neuron-level domain specialization is consistent across samples, far exceeding the ambiguous signals observed at the expert level.
This finding reconciles previous contradictions: expert routing may be driven by tokens or syntax, but neuron sets stably carry domain computation. In other words, expert boundaries are not the endpoint of specialization—neurons are the more reliable unit of observation.
Mixture-of-Experts (MoE) models are sparsely activated neural network architectures composed of multiple "expert" subnetworks and a gating router network. For each input token, the router activates only a few experts (typically 1-2), maintaining low computational overhead despite a massive total parameter count. This design allows MoE models to achieve greater model capacity at lower inference cost, which is why they are widely used in large language models. However, sparse activation also brings interpretability challenges: have experts truly learned to divide labor across different domains or patterns? This is precisely the technical background of the expert specialization debate discussed in the original article.
The difference between expert-level specialization and neuron-level specialization lies in the granularity of observation. Expert-level specialization examines the preference of an entire expert subnetwork for certain types of input, while neuron-level specialization drills down to individual neurons within experts, observing which neurons contribute more to specific domains. Since each expert contains a large number of neurons, expert-level average activation or routing probabilities may mask internal functional differentiation. Neuron-level analysis therefore provides finer-grained and more stable specialization signals.
Training Dynamics: Pretraining Restructures, Post-Training Consolidates
To understand how this neuron-level organization evolves during training, researchers tracked changes in neuron importance across pretraining and post-training for three fully open-source MoE model families.
A key observation is that neuron importance continuously reorganizes across different pretraining checkpoints, while expert-level importance rankings remain relatively stable. This means that even if expert routing patterns appear unchanged, the functional allocation of underlying neurons is still dynamically adjusting. In contrast, the post-training stage largely preserves neuron rankings, indicating that post-training is more about fine-tuning than reshaping.
This stage difference has practical implications for model development: pretraining is the critical window for neuron specialization to form, while post-training tends to consolidate existing structures. To intervene in a model's domain capabilities, one may need to exert influence in the earlier stage.
Pretraining and post-training are the two main stages of large language model training. Pretraining is typically conducted on large-scale unlabeled corpora, using self-supervised learning (such as predicting the next token) to give the model general language abilities and world knowledge; post-training includes instruction fine-tuning, reinforcement learning from human feedback, and other steps to align the model with specific tasks and human preferences. The original article finds that neuron importance continuously reorganizes across pretraining checkpoints but remains largely stable during post-training, meaning the two stages play different roles in shaping the model's internal functional organization: pretraining is the period of structure formation, post-training is the period of structure consolidation. This distinction is instructive for understanding when and how to intervene in model capabilities.
RAMPr Pruning: Balancing Multi-Domain Capabilities
Building on the neuron-level perspective, researchers further proposed RAMP-based Pruning (RAMPr) for selecting neurons across experts to balance the preservation of multi-domain capabilities.
At a 75% neuron retention rate, RAMPr preserved an average of 91-98% of the original full performance across six models. When the retention rate dropped to 50%, the method reduced resident GPU memory by 36-43% and improved batched prefill throughput by 17-39%. These figures demonstrate that neuron-level pruning can significantly compress models while maintaining multi-domain capabilities.
Compared with traditional expert-level or structured pruning, RAMPr's advantage lies in directly targeting domain-specialized neurons, avoiding excessive damage to multi-domain capabilities. The code is open-sourced on GitHub (ydyjya/Neuron-Level-Specialization-in-MoE) for the community to reproduce and extend.
Pruning is a common technique for model compression that reduces model size and computational overhead by removing redundant weights, neurons, or experts. In MoE models, pruning can occur at different granularities: expert-level pruning directly removes entire experts, structured pruning removes attention heads or layers, and neuron-level pruning finely removes individual neurons. RAMPr belongs to neuron-level pruning, and its core advantage is using domain-specialized neurons identified by RAMP to preferentially retain neurons critical to multi-domain capabilities during compression, thereby avoiding the domain capability imbalance that expert-level pruning may cause. Resident GPU memory refers to the parameters and caches that always occupy GPU memory during model inference; batched prefill throughput measures the batch size a model can process per unit time when handling input sequences (the prefill stage). Together, these two metrics reflect the practical deployment benefits of pruning.
Implications for MoE Model Design and Optimization
This study provides new tools for understanding and optimizing MoE models. First, RAMP profiling can be used to diagnose a model's degree of domain specialization, helping developers determine whether expert routing truly corresponds to functional differentiation. Second, RAMPr pruning offers a new approach to model compression: significantly reducing memory and computational overhead while maintaining multi-domain performance.
More importantly, the neuron-level perspective reminds us that the internal organization of MoE models is more complex than expert boundaries suggest. Future model design might directly manipulate neuron-level specialization rather than relying solely on increasing the number of experts. As more open-source MoE models emerge, such fine-grained analysis will become increasingly feasible.
The study also leaves open questions: does neuron specialization change with model scale? Can the RAMP method be extended to other modalities? These await further exploration.
One-line Summary
A new study proposes the RAMP method to reveal neuron-level domain specialization in MoE models and develops the RAMPr pruning technique based on it.
Paragraph Summary
Addressing the controversy over the nature of expert specialization in Mixture-of-Experts (MoE) models, this study proposes the Routing-Aware Magnitude Profiling (RAMP) method, shifting the observation granularity from the expert level to the neuron level. The study finds that domain-specialized neuron sets achieve 58-74% overlap across samples within the same domain, but only 15-24% across domains, indicating that neuron-level specialization is more stable than expert-level signals. Training dynamics analysis shows that neuron importance continuously reorganizes during pretraining but largely consolidates during post-training. Based on this, researchers developed the RAMPr pruning technique, which maintains 91-98% of original performance at a 75% neuron retention rate, and reduces GPU memory by 36-43% while improving prefill throughput by 17-39% at a 50% retention rate. This work reconciles the expert-level specialization controversy and provides new tools for understanding, diagnosing, and compressing MoE models.
Related articles

Gemini 4 Neon Leak: A Potential Strong Contender Against Opus 5.5
A Hacker News thread suggests Gemini 4 Neon looks like an Opus 5.5 competitor. This article unpacks the product positioning behind this community signal, shifts in the top-model competitive landscape, and what it means for developers' model selection strategy.

Claude Fable 5.2's Self-Authored Short Film 'Fable' Sparks Discussion
A short film 'Fable,' credited to Claude Fable 5.2 for script, visuals, and directing, appears on Hacker News. This article analyzes the significance of full-pipeline AI creation, community reactions, and what it means for content creators.

Anthropic IPO Pricing: A Signal of AI Giants Going Public
Anthropic's IPO pricing draws attention. This article analyzes its potential impact on AI industry valuation, commercialization paths, and the competitive landscape, and highlights public-market expectations and challenges for AI companies.