Mixture of Experts

MoE models require large memory to accommodate all parameters but only moderate bandwidth, making them suited to high-capacity unified-memory devices rather than gaming GPUs.

MoE (Mixture of Experts) achieves sparse activation by splitting a large FFN into N smaller FFN experts and using a routing network to decide which experts to activate each time.

Expert pruning typically targets mixture-of-experts (MoE) models, removing redundant or unimportant experts to compress model size.

Existing mixture-of-experts (MoE) models for PDEs generally adopt a homogeneous expert architecture, in which all experts share the same network structure and differ only in parameters.

The V4.1 Flash model contains 384 experts, only a few of which are activated per token; these experts are distributed across 64 cards, 6 per card, plus a clone of one popular chef.

KV-Surgeon's experiments cover dense models and mixture-of-experts models with parameter sizes from 4 billion to 30 billion.

When sustained reasoning workloads exceed 40 million output tokens per day, migrating open-weight mixture-of-experts reasoning models to dedicated bare-metal NVIDIA H200/B200 clusters turns the variable per-token penalty into fixed hardware amortization.

The paper's current experiments are limited to steady-state problems and coarse meshes (about 4,900 vertices), and scalability to transient problems and large-scale meshes has not yet been verified.

The model assigns the solution field to multiple vanilla transformer experts, with each expert focusing on a different spectral scale.

The paper proposes a mixture-of-experts (MoE) architecture based on a learnable Hodge-Laplacian for PDE surrogate modeling.

The paper does not discuss in detail the impact of different expert configurations on performance, and the optimal setting of this hyperparameter may vary by problem type.

The model's gating mechanism takes the mesh itself as input, generates weights for each vertex, and then recombines the outputs of the experts, and the process is fully end-to-end learnable without the need to manually specify spectral partition thresholds.

Active parameters describe the portion involved in computation per token, not the model's download size or the memory required to load it.

Google's official documentation states that all parameters of the 26-billion-parameter MoE model are loaded to maintain inference speed, even though only 4 billion parameters are activated.