Meta's 816B Model Activates Only 5% of Parameters: An Animated Deep Dive into MoE Architecture

Meta's newly released 816B-parameter model has drawn widespread attention, and its core highlight is that each token activates only about 5% of its parameters—thanks to the Mixture of Experts (MoE) architecture. A creator in the Reddit community made an animated explainer video that breaks down how MoE works in an intuitive way. Drawing on the core ideas of that animation, this article takes a deeper look at how MoE enables ultra-large models to stay efficient at inference time.

The Core Idea of MoE: Sparse Activation

Traditional dense models require every token to pass through computation over all parameters during inference. A dense model with 816B parameters means every forward pass must use all 816 billion parameters, making the computational cost extremely high. MoE's breakthrough idea is "sparse activation": splitting the feed-forward network layers in the model into multiple "experts," with each token routed to only a small number of experts for computation.

Meta's 816B model uses only about 5% of its parameters per token, meaning the number of parameters actually involved in computation is on the order of 40B. This is equivalent to using the capacity of an 816B model to store knowledge while using only 40B-level compute for inference. This "large capacity, small compute" characteristic is precisely the fundamental reason why MoE is becoming increasingly popular in ultra-large-scale models.

The feed-forward network (FFN) is the component with the largest parameter count in the Transformer architecture, typically accounting for about two-thirds of the model's total parameters. In a standard Transformer, every token passes through the same FFN, which performs two linear transformations plus a nonlinear activation. MoE's approach is to replace this single FFN with multiple parallel FFN copies (i.e., "experts"), each with independent weights but the same structure. After the Router selects a small number of experts for each token, only the selected experts perform FFN computation, while the remaining experts do not participate at all in that token's forward pass. Therefore, MoE's sparsity is reflected at the computational graph level: although the total parameter count grows linearly with the number of experts, the actual floating-point operations per token are proportional only to the number of activated experts.

Routing Mechanism: Who Decides Where Tokens Go

The key component of MoE is the gating network (also called the Router). For each input token, the Router outputs a set of scores that determine which experts the token should be assigned to. The usual approach is Top-K selection: only the K experts with the highest scores participate in computation (commonly K=1 or K=2), while the rest are not activated.

This design brings an engineering challenge: load balancing. If the Router always assigns tokens to a few "popular" experts, the other experts receive no training and the model's capacity is wasted. Therefore, in actual training, an auxiliary loss is usually introduced to encourage tokens to be distributed evenly across experts. The animated explainer presents this intuitively—tokens are diverted like streams of water to different expert nodes, and the routing strategy determines how evenly the flow is distributed.

Top-K gating and auxiliary loss are the most common combination in MoE training. The gating network is usually a lightweight linear layer that maps the token's hidden state to a score for each expert, and after Softmax, the top K experts are selected. If K=1, each token passes through only one expert; if K=2, the outputs of the two experts are weighted and summed according to the gating scores. Auxiliary loss takes various forms, with the most classic being the load-balancing loss: it penalizes the deviation between the proportion of tokens assigned to experts and a uniform distribution, typically using the sum of squares or variance of each expert's token proportion as the penalty term, multiplied by a small coefficient and added to the total loss. This mechanism does not change the main task objective, but it can significantly alleviate the "winner-takes-all" expert collapse problem.

Why a 5% Activation Rate Is Worth Noticing

From the perspective of scaling laws, the larger the model's parameter count, the stronger its knowledge capacity and expressive ability. But the inference cost of a dense model is proportional to its parameter count, which makes 800B-level dense models almost infeasible to deploy. MoE breaks this linear relationship: the parameter count can continue to expand while the per-token computation remains within a controllable range.

Meta's 816B model runs at an activation rate of about 5%, meaning it can achieve knowledge capacity far beyond that of a 40B model at an inference cost close to that of a 40B dense model. For online service scenarios that require high throughput and low latency, this cost-performance advantage is very obvious. However, it should also be noted that the memory footprint of an MoE model does not decrease proportionally due to sparse activation—all expert parameters still need to reside in memory, which imposes another layer of requirements on hardware deployment.

Scaling Law refers to the empirical rule that model performance (such as loss or downstream task accuracy) improves as a power law with the number of parameters, data volume, and compute. Work such as Chinchilla further points out that, under a given compute budget, the parameter count and the number of training tokens should grow roughly in the same proportion to achieve optimal efficiency. MoE does not violate the scaling law; rather, it changes the correspondence between "parameter count" and "per-token computation": it allows the model to continue expanding along the parameter dimension to improve capacity while keeping per-token computation far below the parameter count. This enables an MoE model, under a fixed inference budget, to have a larger parameter storage space than a dense model with the same amount of compute, thereby gaining an advantage on knowledge-intensive tasks.

The Communicative Significance of the Animated Explainer

The value of this Reddit post lies not only in conveying information about Meta's new model, but also in lowering the barrier to understanding MoE through animation. MoE involves concepts such as routing, sparse activation, and load balancing, and purely textual descriptions can easily leave people confused. The animation visualizes the token routing process, making the abstract mechanism of "only some experts are awakened" perceptible.

For developers and tech enthusiasts interested in large model architecture, this kind of popular-science content helps build intuitive understanding of MoE. As more and more frontier models (including Meta's 816B model) adopt the MoE architecture, understanding its basic principles will become a foundational ability for evaluating model performance, inference costs, and deployment plans.

Conclusion

Meta's 816B model runs at an activation rate of about 5% of its parameters, a typical application of MoE sparse architecture in ultra-large-scale models. The Router determines token routing, load balancing ensures expert utilization, and sparse activation decouples model capacity from inference cost. The animated explainer on Reddit explains this set of mechanisms clearly through visualization and is worth watching.

One-line Summary

Meta's 816B MoE model activates only about 5% of parameters per token, using sparse routing to achieve large-capacity, low-compute inference.

Paragraph Summary

Meta's released 816B-parameter model adopts the Mixture of Experts (MoE) architecture, activating only about 5% of parameters per token, with actual computation on the order of 40B. Drawing on a Reddit animated explainer, the article breaks down MoE's core mechanism: splitting the feed-forward network into multiple experts, with a gating network (Router) using Top-K selection to route each token to a small number of experts, achieving sparse activation. Load balancing is a key training challenge, usually addressed with an auxiliary loss that encourages even expert usage. Sparse activation decouples model capacity from inference cost, but the memory footprint does not decrease proportionally—all expert parameters must still reside in memory. The animated visualization lowers the barrier to understanding and helps developers build intuitive understanding of MoE.