Dense Models vs MoE: Active Parameters, Throughput, and a Practical Selection Guide
MoE decouples total from active parameters, trading large capacity for capability and small activation for inference efficiency.
Using NVIDIA Nemotron 3.5 Lightning as a case study, this article explains how MoE's expert routing mechanism decouples knowledge capacity from per-token compute cost — enabling a 30B-parameter model to activate only ~3B parameters per token. Unlike dense models where compute scales linearly with parameters, MoE trades full VRAM residency of all expert weights for dramatically better throughput at scale. The article also addresses MoE's engineering costs — load imbalance, VRAM dependency on total (not active) parameters, and multi-GPU communication overhead — and offers a clear selection framework: choose MoE when compute and throughput are the bottleneck; choose dense when VRAM or deployment simplicity is the constraint.
A Counterintuitive Question: Why Does a 30B Model Only Activate 3B Parameters?
NVIDIA posed a thought-provoking question in its developer blog: how can a model with 30 billion parameters activate only 3 billion of them per token, while still retaining the capabilities of a large model? The answer lies in the architectural design of the Mixture of Experts (MoE) paradigm. Nemotron 3.5 Lightning is a prime example of this approach.
Behind this question sits one of the most fundamental trade-offs in modern LLM engineering: how do you maximize inference efficiency and reduce deployment costs without sacrificing model capability? Understanding the difference between Dense models and sparse MoE models has become an essential lesson in model selection.
The Core Difference Between Dense Models and MoE
Dense models follow straightforward logic: however many parameters the model has, all of them are used in every forward pass. A 30B dense model engages all 30 billion parameters when processing every single token. This design is simple, predictable, and backed by mature training and inference pipelines — but the cost is that compute scales directly with parameter count. More parameters always means higher inference cost.
MoE models introduce an "expert routing" mechanism. The model contains multiple "expert" sub-networks internally, but each token is only routed to a small subset of those experts for computation. This is what explains the "30B model activating only 3B parameters" phenomenon: the model's total parameter count is large (determining knowledge capacity), while its active parameter count is small (determining the compute cost of a single inference pass).
Two Key Concepts: Total Parameters vs. Active Parameters
When discussing MoE, it's essential to distinguish between two numbers that are easy to conflate:
- Total Parameters: The full size of the model's weights, reflecting its knowledge capacity and upper bound on expressive power.
- Active Parameters: The parameters actually involved in computation when processing a single token — this directly determines compute consumption and latency at inference time.
The core value of MoE is decoupling these two numbers — trading on a large total parameter count for model capability, while using a small active parameter count for inference efficiency. This is precisely why models like Nemotron 3.5 Lightning can maintain large capacity while delivering throughput that far exceeds a dense model of comparable scale.
MoE architecture is not a recent invention. Its theoretical roots trace back to the "Mixture of Experts" framework proposed by Jacobs et al. in 1991. What truly brought MoE to prominence in the LLM space was Google's 2017 Sparsely-Gated MoE paper, followed by the practical deployments of Switch Transformer (2021) and Mixtral 8×7B (2023). Modern MoE LLMs typically introduce expert structures within the FFN (feed-forward network) layers of the Transformer. Each expert is essentially an independent FFN, and a lightweight gating network decides which experts each token is routed to. Nemotron 3.5 Lightning follows this mainstream paradigm: shared attention layers remain dense, while FFN layers are replaced with sparsely-activated expert combinations — achieving an architectural separation between "knowledge storage" and "inference computation."
Throughput: Where MoE Shines
From a throughput perspective, the gains from MoE are substantial. Because each token only activates a fraction of the parameters, MoE models can handle more concurrent requests on the same hardware, or serve larger model capacity within the same latency budget. For production environments requiring high concurrency and large-batch inference, this efficiency advantage translates directly into lower server costs.
That said, MoE comes with real trade-offs. The expert routing mechanism introduces additional complexity: load imbalance can cause some experts to be overloaded while others sit idle; when experts are distributed across multiple GPUs, extra communication overhead is introduced; and loading the model requires keeping all expert weights resident in VRAM — meaning memory footprint is governed by total parameter count, not active parameter count. In other words, MoE saves on compute, but not necessarily on memory.
Load imbalance is one of the most challenging engineering problems in deploying MoE at scale. If the gating network consistently favors a handful of "popular experts," the rest remain underutilized, the model's effective capacity is diminished, and those hot experts become compute bottlenecks. To address this, researchers introduced auxiliary loss functions during training to encourage uniform load distribution across experts, as well as expert capacity limits — when a given expert has already received more tokens than its cap allows, excess tokens are dropped or passed through directly, sacrificing a small amount of accuracy for more controllable routing. In multi-node, multi-GPU deployments, expert parallelism also introduces cross-device All-to-All communication, which is one of the primary reasons MoE tends to have higher latency than a dense model of equivalent compute in distributed settings.
How to Choose Between Dense and MoE
There is no universally superior option — the right choice depends on matching your specific deployment constraints and business objectives. Here are several dimensions to guide your decision:
When to Prefer MoE
- You need high throughput and low per-unit cost for large-scale online serving
- You have ample VRAM but are sensitive to compute latency and concurrency capacity
- You want to maximize model capability within a constrained compute budget
When to Prefer Dense Models
- Your deployment environment is VRAM-constrained and cannot accommodate the full weight footprint of a large MoE model
- You prioritize predictable inference behavior and a simpler deployment pipeline
- At small-to-medium scale, dense models carry lower engineering complexity and are easier to tune
In short: if your bottleneck is compute and throughput, MoE is generally the better solution. If your bottleneck is memory capacity or operational complexity, a dense model is often the safer bet.
Closing Thoughts
The value of the Nemotron 3.5 Lightning example is that it concretely illustrates the engineering philosophy behind MoE — "large capacity, small activation." As model scale continues to grow, MoE offers a compelling path toward balancing capability and cost. But the routing complexity, memory overhead, and communication costs it introduces also demand a more careful evaluation from teams making selection decisions. Understanding the distinction between active parameters and total parameters is the essential first step toward making the right call.
Related articles
LynnReal-Omni: 32B Unified Video Diffusion Model Goes Open Source with Multi-Task Coverage in Four Steps
LynnReal-Omni is a 32B unified video diffusion model on MiniMax H3, covering text-to-video, pose guidance, style transfer, restoration in 4 steps. Flash version generates 540p video in 377ms on one H100.
Anthropic Co-Founder: AI 'Kill Switch' May Need to Be Mandatory by Law
Anthropic's co-founder tells the BBC that AI 'kill switches' may need to be legally mandated. We analyze the industry logic, technical challenges, and the tension between regulation and innovation.
The AI Data Center Boom Is Colliding With Cities Scarred by Heavy Industry
The AI data center boom is clashing with post-industrial communities. Philadelphia's case reveals structural conflicts between AI growth, energy use, water, and environmental justice.