Unverified50% confidenceFactExact time
The Nemotron-H decoder uses 512-expert MoE layers, and at inference time dynamic routing activates only 22 experts, so of the 120B total parameters only 12B participate in computation per step.
1
Sources
50%
Confidence
Long-term
Relevance
10/16/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedThe model has 512 experts per layer, but only 10 are activated per token, meaning the weights actually participating in computation are approximately 6B.70% similarUnverifiedMoE models activate only approximately 10 out of 512 experts per token, with an effective activated parameter count of approximately 20–30B.67% similarUnverifiedThe project adopts a sparse mixture-of-experts (MoE) architecture, in which only about 9B parameters are activated per token out of 74B total parameters, an activation ratio of about 12%.66% similarUnverifiedThe MoE architecture activates only a small subset of expert sub-networks when processing each input token, typically 1/8 to 1/16 of the total number of experts.65% similarUnverifiedThe research team trained a series of decoder-only architecture models from scratch, with parameter scales ranging from 47 million to 349 million.64% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1047833API
curl https://kongchang.com/api/v1/knowledge/claims/1047833MCP
get_claim(id=1047833)