Total Parameter Count Is Obsolete: Understanding the Two-Number Era of MoE Models

Why "Total Parameter Count" Is No Longer a Useful Metric

If you're just getting started with large language models and feeling overwhelmed by various parameter sizes, there's one distinction that matters far more than the headline numbers—it determines how you truly understand a model's capabilities and costs.

For a long time, we've been accustomed to using "total parameter count" to gauge a model's power: 7B, 13B, 70B, 175B… as if bigger numbers automatically meant better performance. But with the proliferation of Mixture-of-Experts (MoE) architecture, a single total parameter count can no longer accurately describe a model's real behavior. Ling 3.0 Tiny, released by Ant Group's research lab, serves as an excellent example, clearly demonstrating just how wide the gap between these two numbers has become.

The concept of Mixture-of-Experts dates back to a 1991 paper by Robert Jacobs et al., but its large-scale application in large language models is a recent development. Google's Switch Transformer (2021) was the first to scale MoE to the trillion-parameter level, proving that sparse activation could dramatically expand model capacity without increasing computational cost. Later, Mistral's Mixtral 8x7B (released late 2023) became the landmark MoE model in the open-source community, using a design with 8 experts and 2 activated per token to match the performance of dense models far larger than its active parameter count. DeepSeek^Ad-V2 and the Qwen series have also adopted MoE architecture. MoE has evolved from an experimental technique to one of the industry's mainstream choices.

Ant Group is an Alibaba-affiliated company with Alipay as its core business, and has been investing heavily in AI infrastructure and large model R&D in recent years. Its AI research lab spans financial risk control, natural language processing, code generation, and more. The Ling series represents Ant's flagship general-purpose LLM effort, positioned for efficient inference scenarios. Choosing to release Ling 3.0 Tiny with MoE architecture reflects a clear industry trend: in deployment-cost-sensitive commercial scenarios (such as real-time responses in financial services, on-device inference), achieving optimal cost-performance by pairing larger knowledge capacity with smaller computational overhead. This aligns with the technical directions taken by DeepSeek, Mistral, and other teams.

The Core Difference Between Dense Models and MoE Models

To understand this issue, we first need to distinguish between two fundamental architectures.

Dense Models: Every Parameter Participates in Computation

A Dense Model works in a straightforward way: for every input token, the model engages all of its parameters in computation. In other words, if a dense model has 8B parameters, all 8B parameters are "working" when processing each token.

This design is simple and predictable, but the trade-off is that the larger the model, the higher the computational cost per token, and the slower it runs. Parameter count and computation are bound together in dense models. Current mainstream dense models include Meta's LLaMA series, Google's Gemma series, and others. They remain highly competitive choices at small to medium scales (7B-70B) because their engineering implementation is simple, inference behavior is predictable, and hardware utilization is easier to optimize.

MoE Models: Only a Subset of Parameters Is Activated

Mixture-of-Experts (MoE) models take a fundamentally different approach. They split the network into many "sub-networks" (i.e., experts) and equip it with a Router. For each token, the router selects only a few experts to participate in computation, while the rest remain "dormant."

The router is the most critical and subtle component in MoE architecture. It's typically a lightweight linear layer that takes the current token's hidden state representation as input and outputs scores (logits) for all experts. Through Top-K selection (e.g., choosing the 2 highest-scoring experts), the router determines which experts process the current token. Selected experts compute independently, then their outputs are combined via weighted summation using the router's assigned weights. Training the router faces a classic challenge—load balancing: if the router consistently assigns tokens to just a few experts, other experts will degrade from insufficient training, creating a "winner-takes-all" situation. To address this, researchers typically introduce auxiliary loss functions to encourage uniform distribution, or employ strategies like expert capacity limits.

This means that while the model possesses a massive total number of parameters, only a small fraction actually participates in each computation. Parameter count and computation are decoupled here. This is precisely how MoE architecture can scale knowledge capacity while maintaining relatively low inference costs.

Two Key Numbers: Total Parameters vs. Active Parameters

Thus, evaluating an MoE model requires two numbers, not one:

• Total Parameters: Primarily determines your memory footprint. Because regardless of whether an expert is activated, its weights must be loaded into GPU memory, ready to be called upon at any time.
• Active Parameters per Token: Primarily determines speed and cost. Because the actual arithmetic workload is determined only by the activated portion of parameters.

Understanding the relationship between total parameters and memory requires some concrete numbers. Taking FP16 (half-precision floating point) format as an example, each parameter occupies 2 bytes. Therefore, a 7.9B parameter model requires approximately 15.8GB of memory for weights alone. FP32 would require 31.6GB, INT8 quantization can compress it to about 7.9GB, and INT4 quantization to approximately 3.95GB. But this is only the static weight footprint—actual inference also requires additional memory for KV cache (which grows linearly with context length), activations, and other intermediate computation results. This is why "total parameters determine whether you can run it at all": even if active parameters are only 1.3B, you still need to load all 7.9B parameters into memory, because the router cannot predict which experts will be selected before each token arrives. Some engineering optimizations like expert offloading can place infrequently used experts in CPU memory or disk for on-demand loading, but this introduces significant latency overhead.

Take Ling 3.0 Tiny as an example: 7.9B total parameters, approximately 1.3B active per token.

This pair of numbers reveals a striking fact: this model carries the "knowledge footprint" of nearly an 8B model, but the arithmetic operations it performs per token are roughly only one-sixth of what an 8B dense model would require. In other words, it has the "brain capacity" of a large model with the "computational speed" of a small one.

The term "knowledge footprint" here refers to the total world knowledge encoded across all of the model's parameters. In dense models, all knowledge is compressed into the same set of parameters, all of which get "consulted" during every inference. In MoE models, different experts can be understood as storing knowledge from different domains or patterns—some experts may be better at mathematical reasoning, others at language translation. The larger the total parameter count, the more facts, patterns, and associations the model can "remember"—this is knowledge capacity. But storing knowledge and retrieving knowledge are two different things: MoE implements a form of "on-demand retrieval" through the router, calling only the subset of knowledge most relevant to the current input, thus avoiding the heavy computational cost that would otherwise be required to access large-capacity knowledge.

This also explains why looking at total parameter count alone can be misleading—an MoE model labeled "8B" may have inference costs far lower than a dense model of the same name; conversely, a model with very few active parameters may have considerable memory requirements.

Cognitive Traps Learners Should Watch Out For

When building cognitive habits around these new metrics, there's one cautious attitude worth developing early: numbers published by vendors are essentially marketing claims until independently reproduced.

Ling 3.0 Tiny's 7.9B total parameters and 1.3B active parameters are officially published data, and no independent third-party evaluations have yet been seen to verify its actual performance. This doesn't mean these numbers are necessarily inaccurate, but rather reminds us: in the absence of independent reproduction, any performance claims should be taken with a grain of salt.

Furthermore, the model is currently API-only. This creates a limitation for learning: you cannot inspect its internal structure, nor can you perform ablation experiments on it.

Ablation experiments are a core methodology in deep learning research for validating design decisions. The basic idea is: systematically remove or replace a component of the model, observe performance changes, and thereby infer that component's actual contribution. For example, to verify the effectiveness of the router in MoE, researchers might replace Top-K routing with random routing to see how much performance drops; to understand the impact of expert count, they might progressively reduce the number of experts and observe the effects. Ablation experiments matter because a model's final performance is the result of all components working together—looking at overall metrics alone cannot attribute contributions. For API-only models, since weights and architectural details are inaccessible, researchers can only perform black-box testing (such as prompt variant experiments) and cannot conduct this kind of fine-grained component-level analysis, which significantly limits the ability to extract transferable engineering insights.

Therefore, even if it's a decent "tool to use," it's not an ideal "object to study"—you can only observe its outputs without being able to dissect its mechanisms.

For learners who want to deeply understand how MoE works, open-source, inspectable models are better research material. Currently available open-source MoE models include Mixtral 8x7B, DeepSeek-MoE, OLMoE, and others, all of which provide complete weights and detailed technical reports, allowing researchers to conduct various experiments and analyses.

Why You Should Care About the Distinction Between Total and Active Parameters

Online explanations of "active parameters vs. total parameters" tend to share a common flaw: they jump straight into routing math formulas while skipping the most critical piece—why a beginner should care about this distinction in the first place.

The answer is quite practical:

1. It determines whether you can even run the model. Total parameter count corresponds to memory requirements—if your hardware can't fit all the parameters, the model simply won't load, regardless of how many it activates. Taking consumer-grade GPUs as an example, an RTX 4090 has 24GB of VRAM, which at FP16 precision can hold at most approximately 12B parameters of model weights (while still needing to reserve space for KV cache and runtime overhead). This means that even if an MoE model's active parameters are only 2B, if total parameters exceed 12B, a single card still can't run it directly without quantization or model parallelism techniques.

2. It determines how much you'll pay and how long you'll wait. Active parameter count corresponds to inference computation, directly affecting API call costs and response latency. In cloud service pricing, computational cost is typically proportional to FLOPs (floating-point operations), and FLOPs per token are roughly linearly related to active parameter count. Therefore, an MoE model activating 1.3B parameters has a per-token inference cost approximately one-sixth that of an 8B dense model—which can save considerable expenses in high-volume calling scenarios.

3. It determines how you compare models horizontally. Comparing a dense model and an MoE model by "total parameters" is like comparing a truck and a drone by "volume"—the numbers aren't wrong, but they completely miss the point. The correct approach is: use active parameter count to benchmark inference speed and cost, use total parameter count to benchmark knowledge capacity and memory requirements, and compare each dimension separately.

Conclusion: From a Single Number to Two-Dimensional Understanding

The rise of MoE architecture signifies that our approach to evaluating models must evolve from "one-dimensional" to "two-dimensional." Total parameter count tells you how much the model "knows"; active parameter count tells you how fast and expensive it is to run. Ling 3.0 Tiny, with its contrast of 7.9B total parameters versus 1.3B active parameters, vividly demonstrates the separation of these two dimensions.

It's worth noting that this "two-dimensional understanding" may only be the starting point. As architectural innovations continue to advance, we may need to pay attention to even more dimensions in the future: how context length affects KV cache memory usage, performance-efficiency trade-offs under different quantization precisions, and how acceleration techniques like speculative decoding impact actual throughput. Model evaluation is evolving from a single scalar to a multi-dimensional vector, and understanding "total parameters ≠ active parameters" is merely the first step in this cognitive upgrade.

The next time you see a model's "parameter count" in a headline, consider asking one more question: is this total parameters, or active parameters? Only when you've clarified this point have you truly understood what that number means.

Key Takeaways