ConceptByte Pair Encoding / BPE

Byte Pair Encoding (BPE) Tokenizer

A subword tokenization algorithm based on byte-pair merging. It iteratively merges high-frequency character pairs to build a vocabulary and is a commonly used tokenization method in mainstream language models such as GPT and LLaMA.

Timeline (last 90 days)

Oct 18

OpenAI's BPE tokenizer averages about 4 characters/token for English, but symbols, indentation, and short identifiers in code pull the characters/token ratio down to 3.5~3.8.

Unverified50%

All Facts (1)

Unverified

OpenAI's BPE tokenizer averages about 4 characters/token for English, but symbols, indentation, and short identifiers in code pull the characters/token ratio down to 3.5~3.8.

50%

Source Articles

Routing Models with a Token Ledger: A 60-Line Python Approach to Cost ControlOct 18 Training a 200M Parameter LLM from Scratch: A Real-World Engineering War StorySep 12