Byte-Level Language Model
A language model that uses raw byte sequences rather than tokenized units as its input and output units. It requires no pretrained tokenizer, can cover arbitrary Unicode characters and binary data, and is suited to multilingual and open-vocabulary settings.
Timeline (last 90 days)
A study systematically examined the impact of text encoding on byte-level language model performance, concluding that UTF-8 is not optimal for all languages.
Byte-level language models use raw byte sequences directly as the input and output units, without relying on a pretrained tokenizer.
Meta published a new paper attempting to let language models skip the tokenizer and read raw bytes (UTF-8 bytes, 0-255) directly as input.
All Facts (3)
A study systematically examined the impact of text encoding on byte-level language model performance, concluding that UTF-8 is not optimal for all languages.
50%UnverifiedByte-level language models use raw byte sequences directly as the input and output units, without relying on a pretrained tokenizer.
50%UnverifiedMeta published a new paper attempting to let language models skip the tokenizer and read raw bytes (UTF-8 bytes, 0-255) directly as input.
50%