unigram-normalized loss
A loss evaluation metric claimed to be tokenizer-independent, which normalizes model loss through a unigram language model. Research finds, however, that its high correlation with BPB stems from confounding factors rather than metric independence.
Timeline (last 90 days)
unigram-normalized loss is not truly tokenizer-agnostic.
The literature previously reported a correlation of up to 0.989 between unigram-normalized loss and BPB, but after correcting for confounders the correlation plummets to 0.057.
Starting from the same data, BPB and unigram-normalized loss select different vocabularies at every model scale.
All Facts (3)
unigram-normalized loss is not truly tokenizer-agnostic.
50%UnverifiedThe literature previously reported a correlation of up to 0.989 between unigram-normalized loss and BPB, but after correcting for confounders the correlation plummets to 0.057.
50%UnverifiedStarting from the same data, BPB and unigram-normalized loss select different vocabularies at every model scale.
50%