The Three Great Tokenizer Myths: Is BPB Really Reliable?

Introduction: The Tokenizer Determines the World the Model Sees

In the modern language model pipeline, the tokenizer is often the component that gets fixed first and scrutinized the least. Before the model even touches the data, the tokenizer has already determined the granularity at which that data is presented. A study titled Tokenizer Myths and Where To Find Them takes direct aim at this overlooked component, systematically challenging three widely accepted "myths" in the tokenizer research literature. These myths affect not only academic evaluation but also subtly shape the practice of product selection and model comparison.

Myth One: BPB Is Independent of the Tokenizer?

Bits-Per-Byte (BPB) has long been regarded as a "tokenizer-agnostic" evaluation metric, on the grounds that it is computed as tokens per byte multiplied by bits per token. The researchers point out that these two factors do not cancel each other out. Using experiments with untrained models, they show that BPB actually favors certain tokenizers over others—the model itself has learned nothing, yet the tokenizer-model combination exhibits differences.

This means that when researchers use BPB to compare different tokenization schemes, they may simply be comparing the structural preferences of the tokenizers themselves rather than model capability. This finding is a direct warning to practitioners who rely on BPB for tokenizer selection.

BPB (Bits-Per-Byte) is an evaluation metric that normalizes a language model's loss to the byte level. It is typically computed by first dividing the model's output negative log-likelihood per token (in bits) by the number of bytes corresponding to that token, then averaging over the entire dataset. The original intent is to eliminate the influence of different vocabulary sizes and segmentation granularities across tokenizers, so that the compression efficiency of models under different tokenization schemes can be compared "fairly." However, BPB's denominator is the byte count while its numerator is the token-level loss, and both depend on how the tokenizer segments the text. When a tokenizer merges more characters into a single token, the number of bytes carried per token increases, but the difficulty of the model predicting that token may also change in tandem—the two factors do not necessarily cancel out. Thus, BPB's "tokenizer-agnostic" label is more of an idealized assumption than a mathematically rigorous property.

Myth Two: Lower BPB Means a Better Model?

The second myth is more fundamental: does lower BPB mean a better model? The research answers in the negative. In untrained models, BPB strongly favors certain vocabulary sizes even when model accuracy remains at chance level; in real models trained on the same data, the correlation between BPB and downstream task performance is weak.

In other words, as a compression-efficiency metric, BPB is clearly disconnected from the model's usefulness on actual tasks. Using BPB as a single proxy for model quality may lead researchers astray.

Untrained models are often used in tokenizer research as controlled probes: their parameters are randomly initialized and have learned no linguistic regularities from data, so their output distribution is determined solely by the model architecture, initialization, and tokenizer. If an evaluation metric already exhibits a systematic preference for a particular tokenizer or vocabulary size on untrained models, it indicates that the differences measured by that metric do not come from model capability but from the interaction between the tokenizer and the model structure. The logic of this experimental design is similar to a controlled experiment: hold the variable of "learning" fixed at zero, and observe whether the evaluation metric still varies with the tokenization scheme. If the variation is significant, the metric cannot be regarded as a measure that purely reflects model quality.

Myth Three: Do "Tokenizer-Agnostic" Metrics Agree With Each Other?

The third myth concerns the consistency among so-called "tokenizer-agnostic" metrics. The study specifically examines "unigram-normalized loss" and points out that it is not truly tokenizer-agnostic. The literature had reported a correlation as high as 0.989 between this metric and BPB, but the researchers found that this high correlation came from a confounding factor; after correcting for that confounder, the correlation dropped sharply to 0.057. Starting from the same data, the two metrics selected different vocabularies at every model scale.

This result shows that different "agnostic" metrics may yield contradictory conclusions, and the choice of metric itself determines the direction of the research question.

Confounding factors in statistical and evaluation research refer to third variables that simultaneously affect both the independent and dependent variables, thereby creating spurious correlations. In the context of tokenizer evaluation, a typical confounder might be vocabulary size or token frequency distribution: it affects both the value of BPB and the value of unigram-normalized loss, causing the two metrics to appear highly consistent when this factor is not controlled. Once the factor is fixed or removed through stratification, regression adjustment, or experimental design, the previously high correlation of 0.989 drops sharply to 0.057, indicating that the earlier high correlation did not stem from the two metrics conceptually measuring the same thing but from their sharing the same confounding source. Identifying and correcting for confounders is a key step in judging whether a "tokenizer-agnostic" metric is truly independent.

The Tokenizer Trilemma: Choosing a Metric Is Choosing a Research Question

Based on these findings, the authors propose a "tokenizer trilemma": no metric can simultaneously satisfy three conditions—assigning the highest score only to the true distribution, giving the same score to the same prediction, and making all untrained models tie. Therefore, choosing an evaluation metric is essentially choosing the research question you want to answer.

This framework elevates tokenizer evaluation from a technical detail to a methodological level. For researchers and engineers reading the tokenizer literature, the authors offer a clear caution: when encountering conclusions such as "tokenizer-agnostic" or "lower BPB," one needs to ask about the assumptions and confounders behind the metric.

Implications for Practice

The value of this research lies not in refuting BPB or existing metrics but in revealing their respective boundaries of applicability. For teams engaged in model evaluation, tokenizer selection, or vocabulary design, there are at least three actionable recommendations: first, do not use BPB as the sole criterion for model quality—combine it with downstream task performance; second, remain cautious about metrics claimed to be "tokenizer-agnostic" and pay attention to their corrected correlations; third, when comparing different tokenization schemes, be clear about which research question your chosen metric corresponds to.

The tokenizer may be small, but it determines the first interface between the model and the data. Understanding these myths is the first step toward more reliable evaluation.

One-line Summary

Research reveals that metrics such as BPB are not tokenizer-agnostic—choosing a metric is choosing a research question.

Paragraph Summary

A study titled Tokenizer Myths and Where To Find Them systematically challenges three popular myths in tokenizer evaluation. First, BPB is not tokenizer-agnostic: on untrained models, BPB already favors certain tokenizers—the model has learned nothing, yet the tokenizer-model combination already produces differences. Second, low BPB does not equal a good model: on untrained models, BPB favors certain vocabulary sizes while accuracy remains at chance level, and in real models BPB correlates weakly with downstream performance. Third, unigram-normalized loss is not truly tokenizer-agnostic—its high 0.989 correlation with BPB comes from a confounder and drops sharply to 0.057 after correction. Based on this, the authors propose a "tokenizer trilemma," pointing out that no metric satisfies all three ideal conditions simultaneously and that choosing an evaluation metric is essentially choosing a research question, and they offer practical recommendations such as using BPB with caution and paying attention to corrected correlations.