Unverified50% confidenceBenchmarkExact time
With the expert pool expanded from 8 to 128 experts, 4 experts selected per token, and activated parameters held constant at about 3.2 billion, the total parameter capacity grows from 4.6 billion to 47 billion (a tenfold expansion), while training throughput drops by less than 5%.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/14/2026
First Seen
Valid until: 1/12/2027
Sources
Related Entities
Related Claims
UnverifiedThe model has 512 experts per layer, but only 10 are activated per token, meaning the weights actually participating in computation are approximately 6B.73% similarUnverified对LLaMA-7B所有注意力层应用r=8的LoRA，可训练参数量从70亿降至约400万，压缩比超过1750倍70% similarUnverifiedTaking a 1000×1000 weight matrix as an example, if the LoRA rank is set to 8, only approximately 16,000 parameters need to be trained instead of the original 1,000,000 parameters.68% similarUnverifiedInvoking multiple experts simultaneously causes token consumption to multiply, with total token consumption growing linearly or even super-linearly with the number of experts68% similarVerifiedFor a 7-billion-parameter model, LoRA typically requires training less than 1% of the total parameters, and GPU memory requirements can be reduced by more than 60%.65% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1025297API
curl https://kongchang.com/api/v1/knowledge/claims/1025297MCP
get_claim(id=1025297)