Unverified50% confidenceFactExact time
The 32k vocabulary collapsed at the 50M scale because its embedding matrix occupied nearly half of the model parameters and could not be adequately trained within 1,000 training steps.
1
Sources
50%
Confidence
Long-term
Relevance
10/13/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedIf samples exceeding 32K tokens rarely appear in a model's pre-training corpus, the model's performance in that range tends to be unstable even when the context window is extended through positional encoding.77% similarUnverifiedNaïve post-training quantization causes individual weight errors to accumulate and amplify across multiple layers—a phenomenon known as compounding error—and at 2-bit or 1-bit precision the model structure collapses70% similarUnverified训练时不能一开始就追求快，若直接压步数模型容易陷入死循环反复生成重复字符，只做第一步的版本碰上难题会卡死，需先练准再练快69% similarUnverified随着模型规模从百万参数扩展至千亿参数，不理解底层数学的工程师越来越难以诊断训练不稳定、梯度爆炸或模式崩塌等问题67% similarVerifiedThe closer an event is to the cutoff date, the smaller its share in the training corpus, and the less clearly the model has mastered it.67% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1019330API
curl https://kongchang.com/api/v1/knowledge/claims/1019330MCP
get_claim(id=1019330)