Pre-tokenization
The first step in a large language model tokenization pipeline, in which deterministic regular expression rules split raw text into non-overlapping segments, establishing boundary constraints for subsequent tokenization algorithms such as BPE or Unigram.
Timeline (last 90 days)
Pre-tokenisation rules constrain which text segments can serve as prediction units, and the compression cost they introduce is masked when comparing tokenizers under the same boundary rules.
A vocabulary that pursues maximum compression is not necessarily the optimal choice for predictive modeling, and there is a tension between compression efficiency and prediction quality.
All Facts (2)
Pre-tokenisation rules constrain which text segments can serve as prediction units, and the compression cost they introduce is masked when comparing tokenizers under the same boundary rules.
50%UnverifiedA vocabulary that pursues maximum compression is not necessarily the optimal choice for predictive modeling, and there is a tension between compression efficiency and prediction quality.
50%