Amazon HOPE Study: Qwen3.5 Model Retains Coding Performance After Halving Experts

Amazon recently published its HOPE research along with accompanying code. One experiment shows that after removing roughly half of the experts from the Qwen3.5-35B-A3B model, its coding performance remained comparable to the original model. This finding offers new empirical reference for model compression and efficient inference in Mixture-of-Experts (MoE) architectures.

What Is the HOPE Research?

HOPE is a research project recently released by Amazon, centered on exploring model architecture optimization and efficiency improvements. Unlike many efforts that only publish papers, Amazon shared both the HOPE research documentation and runnable code, making it easier for the community to reproduce and further explore.

Based on public information, HOPE focuses on how to reduce inference costs or adjust model structure while preserving model capabilities. What sparked discussion this time is the expert pruning experiment targeting the Qwen3.5-35B-A3B model.

Why Does Coding Performance Remain Similar After Removing Half the Experts?

The experiment targets Qwen3.5-35B-A3B, a model that uses a Mixture-of-Experts architecture. The defining feature of MoE is that only a subset of experts is activated during inference, thereby expanding model capacity while controlling computation.

In the experiment, Amazon removed approximately half of the model's experts and then evaluated it on coding tasks. The results showed that the model's coding performance was comparable to the original version. This suggests that for specific coding tasks, the experts in the model may be highly redundant, or that some experts contribute little to coding ability.

It is worth noting that this conclusion is limited to coding performance in "one test." Coding tasks tend to be structured and pattern-based, and a model may rely on just a few key experts to perform them. Whether other types of tasks (such as reasoning or multilingual understanding) remain equally stable is not addressed in the public information.

The core mechanism of the Mixture of Experts (MoE) architecture is "sparse activation": the model contains a router (gating network) and multiple expert sub-networks. During each forward pass, the router computes weights for each expert based on the input token's features and activates only the few experts with the highest scores (typically top-k, such as top-2). This allows the model's total parameter count to be large while keeping the activated parameter count per inference small, thereby expanding model capacity without significantly increasing computational cost. In the name Qwen3.5-35B-A3B, "A3B" refers to an activated parameter count of approximately 3B (3 billion), while the total parameter count is 35B. Expert pruning involves removing some experts after training and adjusting the router accordingly, to reduce total memory footprint and potential communication overhead. Since only a few experts are used per inference, experts that are consistently under-activated or highly overlapping in function become pruning candidates.

Implications for MoE Model Practice

MoE models have been widely adopted in recent years, with the advantage of achieving large model capacity with a relatively small activated parameter count. However, the issue of expert redundancy has long been discussed: are all post-training experts indispensable?

This HOPE experiment provides a concrete case. If coding performance is maintained after removing half the experts, then in practical deployment, trimming experts for specific scenarios could be a viable approach to reducing memory footprint and inference overhead. However, such trimming requires careful evaluation per task type and cannot be simply generalized to all scenarios.

From an engineering perspective, Amazon's release of the code lowers the barrier to verification. Developers can use the HOPE code to test similar expert pruning strategies on their own data and tasks and determine whether they suit their business needs.

Expert redundancy is repeatedly discussed in MoE research due to the "routing collapse" phenomenon during training. Studies have found that the router exhibits self-reinforcing positive feedback: experts that are activated more often receive more gradient updates, become more capable, and are then selected more frequently, ultimately causing a few experts to bear most of the computation while the rest become increasingly "idle." In addition, different experts may be highly similar in representation space, with obvious functional overlap. This means that removing low-contribution experts may have limited impact on model output, which is also the theoretical basis for pruning's feasibility. However, expert importance is closely related to task distribution—the division of labor among experts trained on general corpora may not match the optimal division for a particular vertical task, so pruning effectiveness must be empirically validated on the target task.

Community Reaction and Open Questions

The research sparked discussion on Reddit. Community concerns include: how the pruned model performs on other benchmarks, whether expert removal affects the model's generalization ability, and whether this method applies to other MoE models.

Currently, public information is limited; experimental details, evaluation benchmarks, and specific configurations have not been fully elaborated in the original abstract. Therefore, it is safer to regard this result as "preliminary evidence of expert redundancy in coding tasks" rather than a general conclusion.

For developers focused on model compression and efficient inference, the HOPE code is worth further study. Reproducing and extending the experiments on one's own tasks is a key step in judging the method's practical value.

One-line Summary

Amazon released the HOPE study, showing that Qwen3.5-35B-A3B retains similar coding performance after removing about half its experts.

Paragraph Summary

Amazon published its HOPE research along with accompanying code. One experiment drew attention: after removing roughly half the experts from the MoE model Qwen3.5-35B-A3B, its coding performance remained comparable to the original version. The article introduces the background of HOPE's focus on model architecture optimization and inference efficiency, analyzes the expert redundancy phenomenon this result may reflect, and emphasizes that the conclusion is based on only one coding test and cannot be directly generalized to other tasks such as reasoning or multilingual understanding. It further discusses expert pruning as a viable approach for reducing memory footprint and inference overhead in specific scenarios, notes that open-sourcing the code lowers the barrier for community reproduction and verification, and cautions that the impact of pruning on generalization ability and benchmark performance still awaits more experimental confirmation.