Byte-Level Modeling
A language modeling approach that bypasses the tokenizer and directly predicts over raw UTF-8 encoded bytes as the input sequence; the vocabulary is extremely small (only 256), with no out-of-vocabulary problem, but sequence length increases significantly.
Timeline (last 90 days)
Byte-level models have an extremely small vocabulary (only 256), have no out-of-vocabulary problem, and handle cross-lingual and symbolic processing more uniformly.
Byte-level modeling reduces text granularity to individual UTF-8 bytes, shrinking the vocabulary to 256.
Byte-level modeling eliminates the linguistic imbalance introduced by tokenization, but at the cost of significantly increased sequence length, placing greater demands on the efficiency of the attention mechanism.
All Facts (3)
Byte-level models have an extremely small vocabulary (only 256), have no out-of-vocabulary problem, and handle cross-lingual and symbolic processing more uniformly.
50%UnverifiedByte-level modeling reduces text granularity to individual UTF-8 bytes, shrinking the vocabulary to 256.
50%UnverifiedByte-level modeling eliminates the linguistic imbalance introduced by tokenization, but at the cost of significantly increased sequence length, placing greater demands on the efficiency of the attention mechanism.
50%