Shadow Tokenizer
Drawing on the idea of shadow models, a tokenizer trained on known substitute corpora, used to establish a reference distribution linking token presence to corpus frequency in order to calibrate attack signals.
Timeline (last 90 days)
The RTAM-Fusion method uses shadow tokenizers to calibrate the quality of rare tokens and fuses signals across rarity tiers.
The idea of shadow tokenizers is borrowed from shadow models: the attacker trains several shadow tokenizers on known substitute corpora using a pipeline similar to the target tokenizer, and observes the occurrence patterns of rare tokens to establish a reference distribution.
All Facts (2)
The RTAM-Fusion method uses shadow tokenizers to calibrate the quality of rare tokens and fuses signals across rarity tiers.
50%UnverifiedThe idea of shadow tokenizers is borrowed from shadow models: the attacker trains several shadow tokenizers on known substitute corpora using a pipeline similar to the target tokenizer, and observes the occurrence patterns of rare tokens to establish a reference distribution.
50%