Unverified50% confidenceFactExact time
The 2,103-token vocabulary is directionally wrong: at the 10M scale, the 2k vocabulary has a bpb about 19% higher than the 8k one.
1
Sources
50%
Confidence
Long-term
Relevance
10/13/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedVocabulary choice matters more than backbone size: 10M labels with an 8k vocabulary (8.0M total parameters, bpb 1.168) beats 50M labels with a 2k vocabulary (33.5M total parameters, bpb 1.436) — 4.2 times fewer parameters but 18.7% lower bpb.78% similarUnverifiedOn the held-out set, the 8k–16k vocabularies had the lowest bpb across all three model sizes, while the 2k vocabulary consistently performed worst.74% similarUnverifiedIn the task of completing the last Token of an FP8 quantization function, after DeepSeek-V4-Flash-Base adds equal-sign decorations before the code, when the number of equal signs modulo 4 is 0 or 1 the error rate is as high as 71.3%, and when the remainder is 2 or 3 the accuracy returns to 91.5%.65% similarUnverifiedAll incorrect distractor counting answers were without exception overestimates, and the token counts of incorrect answers were clearly higher than those of correct answers: for gpt-5.4-mini at high, the median token count of incorrect answers was 412 vs 91.5 for correct answers; for gpt-oss-120b, incorrect answers were 5,776 tokens vs 747 tokens for correct answers.64% similarUnverifiedIn the experiment, vocabulary sizes of 2,000, 8,000, 16,000, and 32,000 corresponded to tokens per byte (fertility) of 0.4475, 0.3667, 0.3435, and 0.3275, respectively.63% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1019333API
curl https://kongchang.com/api/v1/knowledge/claims/1019333MCP
get_claim(id=1019333)