tokenizer

中文字符在不同tokenizer下的Token数量差异可达30%-50%

Different models use different tokenization schemes (such as BPE, SentencePiece, etc.), and the same piece of text may be counted as a different number of tokens across different models.

Using another vendor's tokenizer library to compute the target model's token count leads to systematic bias, because different models use different vocabularies, and this bias is unobservable from the caller's perspective.

There is no universal conversion ratio between the character count of a repository and the model's token count; the ratio is model-dependent and drifts with language and coding style.

Changes to the tokenizer's vocabulary size, special token definitions, or context handling methods will cause some preprocessing and postprocessing logic to need updating.

Token segmentation logic varies by model; the handling of Chinese, code, and punctuation each differs, and a passage of English text may be split into far more tokens than the number of words

human-tps uses real tokenizers rather than counting by character or word to measure token speed.

Using real tokenizers for measurement places human t/s and model t/s in a comparable context.

In Apple's dictation technical pipeline, speech first passes through a tokenizer acting as an encoder that maps short-time waveform windows into representations readable by the language model, which then processes these representations.

In on-device scenarios, the encoder's parameter count directly affects power consumption and latency—more parameters means greater DRAM usage, which impacts both energy consumption and response speed.

The always-on tokenizer must compete for the same memory resources as the activated experts, becoming a non-negligible static overhead.

同一段Python代码在不同tokenizer下的Token数差异可达30%-50%