[KongchangAI]
Conceptbyte-level language model

Byte-Level Language Model

A language model that uses raw byte sequences rather than tokenized units as its input and output units. It requires no pretrained tokenizer, can cover arbitrary Unicode characters and binary data, and is suited to multilingual and open-vocabulary settings.

Timeline (last 90 days)

Nov 11

A study systematically examined the impact of text encoding on byte-level language model performance, concluding that UTF-8 is not optimal for all languages.

Unverified50%
Nov 11

Byte-level language models use raw byte sequences directly as the input and output units, without relying on a pretrained tokenizer.

Unverified50%
Oct 24

Meta published a new paper attempting to let language models skip the tokenizer and read raw bytes (UTF-8 bytes, 0-255) directly as input.

Unverified50%

All Facts (3)

Source Articles