BPB
Bits-Per-Byte, an evaluation metric that normalizes language model loss by byte, commonly used to compare the compression efficiency of models across tokenizers, but shown not to be truly tokenizer-independent.
Timeline (last 90 days)
For teams working on model evaluation, tokenizer selection or vocabulary design, it is recommended not to use BPB as the sole criterion for model quality, but to combine it with downstream task performance.
BPB (Bits-Per-Byte) has long been regarded as a tokenizer-agnostic evaluation metric.
Through experiments on untrained models, the researchers show that BPB actually favors certain tokenizers over others.
BPB is computed as tokens per byte multiplied by bits per token, and these two factors do not cancel each other out.
The study gives a negative answer: lower BPB does not mean a better model.
On untrained models, BPB strongly favors certain vocabulary sizes, even when model accuracy remains at chance level.
In real models trained on the same data, the correlation between BPB and downstream task performance is weak.
The literature previously reported a correlation of up to 0.989 between unigram-normalized loss and BPB, but after correcting for confounders the correlation plummets to 0.057.
Starting from the same data, BPB and unigram-normalized loss select different vocabularies at every model scale.
Compared with perplexity, BPB is insensitive to tokenization, making cross-model comparisons easier.
All Facts (10)
For teams working on model evaluation, tokenizer selection or vocabulary design, it is recommended not to use BPB as the sole criterion for model quality, but to combine it with downstream task performance.
50%UnverifiedBPB (Bits-Per-Byte) has long been regarded as a tokenizer-agnostic evaluation metric.
50%UnverifiedThrough experiments on untrained models, the researchers show that BPB actually favors certain tokenizers over others.
50%UnverifiedBPB is computed as tokens per byte multiplied by bits per token, and these two factors do not cancel each other out.
50%UnverifiedThe study gives a negative answer: lower BPB does not mean a better model.
50%UnverifiedOn untrained models, BPB strongly favors certain vocabulary sizes, even when model accuracy remains at chance level.
50%UnverifiedIn real models trained on the same data, the correlation between BPB and downstream task performance is weak.
50%UnverifiedThe literature previously reported a correlation of up to 0.989 between unigram-normalized loss and BPB, but after correcting for confounders the correlation plummets to 0.057.
50%UnverifiedStarting from the same data, BPB and unigram-normalized loss select different vocabularies at every model scale.
50%UnverifiedCompared with perplexity, BPB is insensitive to tokenization, making cross-model comparisons easier.
50%