Unverified50% confidenceFactExact time
Under untruncated training (T=1), the 9th-generation GPT-2 model at a 13000-word budget has 24% fewer vocabulary types than the 0th generation, but its collision entropy is 0.91 bits higher.
1
Sources
50%
Confidence
Long-term
Relevance
11/9/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedAt a training scale approaching 100 billion tokens, the difference in final smoothed training loss between the polynomial version and the native implementation falls between -0.107 and +0.079.68% similarUnverifiedExperiments conducted by the research team on models such as GPT-2 showed that after only 5–9 generations of recursive training, the quality of model outputs degrades significantly.68% similarUnverifiedThe trained model achieves a per-token negative log-likelihood (NLL) of 0.93, compared to 12.60 for a randomly initialized twin model of the same architecture.67% similarUnverifiedThe figure "pretraining throughput reaching 170% of a single H100" is jointly affected by batch size, sequence length, model scale, precision format, and interconnect bandwidth, and is not a general conclusion; the ratio may change with a different model or hyperparameters.66% similarUnverifiedLlama 2 used the AdamW optimizer (beta1 = 0.9, beta2 = 0.95, weight decay 0.1, gradient clipping 1.0), and was pretrained on 2 trillion tokens.65% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1149949API
curl https://kongchang.com/api/v1/knowledge/claims/1149949MCP
get_claim(id=1149949)