Tokka-Bench
An open-source multilingual tokenizer evaluation framework covering 100 natural languages and 20 programming languages. It introduces five complementary evaluation dimensions—bytes per token, unique token coverage, fertility rate, word split rate, and vocabulary composition—for cross-comparing the quality and efficiency of tokenizers across large language models.
Timeline (last 90 days)
Tokka-Bench is an open-source multi-metric tokenizer evaluation framework covering 100 natural languages and 20 programming languages.
Tokka-Bench proposes five evaluation dimensions: bytes per token, unique token coverage, fertility, word split rate, and vocabulary composition.
Tokka-Bench finds that a tokenizer's vocabulary allocation strategy is more important than the raw vocabulary size.
The Tokka-Bench framework, datasets, and an interactive dashboard are all publicly available.
All Facts (4)
Tokka-Bench proposes five evaluation dimensions: bytes per token, unique token coverage, fertility, word split rate, and vocabulary composition.
50%UnverifiedTokka-Bench finds that a tokenizer's vocabulary allocation strategy is more important than the raw vocabulary size.
50%UnverifiedTokka-Bench is an open-source multi-metric tokenizer evaluation framework covering 100 natural languages and 20 programming languages.
50%UnverifiedThe Tokka-Bench framework, datasets, and an interactive dashboard are all publicly available.
50%