Tokenizer Leaking Training Data? Membership Inference Attacks on Rare Tokens in BPE

Introduction: Tokenizers Are More Than Just Preprocessing Tools

In most machine learning pipelines, the tokenizer is treated as a fixed preprocessing component—tokenize first, train later—and few people pay attention to the information it carries itself. But a recent study raises a concerning point: BPE (Byte Pair Encoding) tokenizers are trained on text, and their vocabularies may retain traces of the training corpus. Based on this, the researchers try to answer a question: Can we determine whether a candidate dataset was involved in training a BPE tokenizer by analyzing the tokenizer itself?

The core observation of this study is simple yet powerful: rare strings only become independent tokens when their corpus frequency is high enough to cross the merge threshold. This means membership signals are more likely to be hidden in the rare tokens used by the candidate dataset, rather than in the merge order itself.

BPE (Byte Pair Encoding) is a statistical subword tokenization algorithm, originally proposed by Gage in 1994 for data compression. Its training process starts from a character-level vocabulary, repeatedly counts the co-occurrence frequency of adjacent symbol pairs in the corpus, and merges the most frequent pair into a new token until the preset vocabulary size or number of merges is reached. Therefore, each merge operation corresponds to a "merge rank"—the earlier the rank, the more frequently the symbol pair appeared in the training corpus.

In NLP practice, BPE was introduced to neural machine translation by Sennrich et al. in 2016 and has since become the standard tokenization scheme for mainstream models such as GPT and RoBERTa. The key point is: the formation of the vocabulary is entirely driven by the frequency distribution of the training corpus—which strings become independent tokens and in what order they are merged are directly influenced by the corpus. This is exactly the foundation on which tokenizer membership inference attacks rest—the vocabulary itself encodes information about the training data.

Threshold Model: Why Rare Tokens Become "Fingerprints"

The study formalizes the above intuition into a threshold model of BPE inclusion. Under this model, the likelihood ratio test assigns the greatest weight to tokens that are rare in the background corpus but frequent in the candidate dataset. This makes sense: if a token is rare across the entire vocabulary but happens to be heavily used by the candidate dataset, it likely crossed the merge threshold due to that dataset's contribution.

A further inference is: once the presence of a token and the dataset frequency are known, the merge rank and merge-graph position should no longer provide first-order membership signals. In other words, what leaks information is not "the order of merges" but "which rare tokens were retained." This conclusion has direct implications for designing defenses—hiding only the merge order is insufficient to block leakage.

Membership Inference Attack aims to determine whether a specific data sample or dataset appeared in the training set of a target model, and is a classic privacy attack paradigm. Traditional research mostly targets model weights or output probabilities (e.g., loss- or confidence-based attacks), whereas this paper shifts the focus to the tokenizer level.

The privacy implication is: if an attacker can confirm that a sensitive dataset was involved in tokenizer training, they can infer the institution's data collection and processing behavior, and even trace back to the corpus source. The core statistical tool used in this paper is the Likelihood Ratio Test: under the two hypotheses of "candidate set participated in training" and "did not participate in training," the probability of observing the token distribution is calculated separately, and the ratio is used as the basis for discrimination. Rare tokens that are frequent in the candidate set receive the greatest weight precisely because their probability difference under the two hypotheses is the most extreme, thus providing the strongest discriminative power.

RTAM-Fusion: Calibrating Rare Token Quality with Shadow Tokenizers

Based on the threshold model, the authors propose the RTAM-Fusion method. It uses shadow tokenizers to calibrate the quality of rare tokens and fuses signals across rarity levels.

On the tokenizer membership inference benchmark, RTAM-Fusion outperforms the Vocabulary Overlap baseline: both AUC and balanced accuracy improve, while requiring fewer shadow tokenizers. This shows that focusing on the presence signals of rare tokens is more effective than simply comparing vocabulary overlap. The study also points out that at production-scale vocabulary sizes, existing defenses still leak membership signals against adaptive attackers—meaning deployed tokenizers may be more "talkative" than expected.

Shadow Tokenizers borrow the idea from shadow models: the attacker trains several "shadow" tokenizers on known substitute corpora using a process similar to the target tokenizer, observes the occurrence patterns of rare tokens under these controlled conditions, and thereby establishes a reference distribution of "token presence—corpus frequency." Since the target tokenizer's training corpus is inaccessible, shadow tokenizers provide a calibratable control baseline.

AUC (Area Under the ROC Curve) measures the overall ability of a discriminator to rank positive examples above negative examples; a value of 0.5 corresponds to random guessing, and the closer to 1 the better. Balanced Accuracy is the average of positive and negative recall, and better reflects true performance than ordinary accuracy when classes are imbalanced. The comparison baseline Vocabulary Overlap in this paper only counts the literal overlap between the candidate set and the vocabulary, ignoring frequency and rarity information. Therefore, the improvement of RTAM-Fusion demonstrates that "which rare tokens are retained" is more discriminative than "how many words overlap."

Implications for Data Privacy and Model Governance

This work reveals a long-overlooked privacy attack surface of tokenizers: even if model weights are protected, the tokenizer itself can become a leakage channel for training data membership information. For institutions that train tokenizers on large-scale corpora, the risks of releasing vocabularies need to be reassessed.

At the same time, the study suggests a shift in defense thinking—rather than trying to conceal the merge process, it is better to focus on the distribution characteristics of rare tokens. If noise can be introduced during tokenizer training or rare tokens can be generalized in the future, the feasibility of membership inference may be reduced. The code has been released in an anonymous repository for reproduction and further verification.

Conclusion

Tokenizers are not neutral preprocessing tools; their vocabularies record frequency traces of the training corpus. RTAM-Fusion achieves better membership inference performance through rare token signals and reveals the persistent leakage problem of existing defenses at production-scale vocabularies. This perspective deserves continued attention in the fields of data privacy and model security.