Unverified50% confidenceFactExact time
Starting from the same data, BPB and unigram-normalized loss select different vocabularies at every model scale.
1
Sources
50%
Confidence
Long-term
Relevance
11/4/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedThe literature previously reported a correlation of up to 0.989 between unigram-normalized loss and BPB, but after correcting for confounders the correlation plummets to 0.057.73% similarUnverifiedThe validation loss on preference data measures how well the model fits the preferred responses, whereas downstream benchmarks examine generation quality; the two are not equivalent, and the model may reduce validation loss while degrading on downstream tasks.66% similarUnverifiedThe study gives a negative answer: lower BPB does not mean a better model.66% similarUnverifiedMost embedding models are trained using contrastive loss (InfoNCE or its variants), which only cares about the relative ranking of positive samples and not the absolute value of scores.66% similarUnverifiedFocal Loss adds a modulating factor (1-p_t)^γ before the standard cross-entropy loss, with γ typically set to 2, causing the model to focus on hard examples.65% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1134081API
curl https://kongchang.com/api/v1/knowledge/claims/1134081MCP
get_claim(id=1134081)