o200k_base
A tokenizer standard released by OpenAI with a vocabulary of approximately 200,000 tokens. Introduced alongside the GPT-4o model series, it serves as the vocabulary used by current mainstream frontier models and offers improved encoding efficiency for code symbols and multi-byte characters compared to its predecessor, cl100k_base.
Core Facts
Timeline (last 90 days)
The authors reproduced the previously published "The Tokenizer Tax" table, with all 66 of 66 cells within 0.5% error; the average tax rate is 8.03x under cl100k_base (8.0x in the original paper) and 2.14x under o200k_base (2.1x in the original paper).
The author's measurements show that the JSON Schema definitions of 18 tools consume 2,388 tokens, averaging 133 tokens per tool, and in a 10-step Agent run they can account for 36% of input tokens and 32% of the total bill.
o200k_base is the tokenization encoding used by newer GPT series, with a larger vocabulary, and its segmentation efficiency for structured text such as code and JSON is usually better than the older cl100k_base.
Encoded with o200k_base, 'I love strawberry' becomes [40, 3047, 101830].
o200k_base is the tokenizer OpenAI introduced for the GPT-4o series, with a vocabulary of about 200,000, and its compression efficiency for multilingual (especially non-English) text is clearly better than the earlier cl100k_base.
In a measured sample of 20 publicly available German-language pages, the median page's raw HTML was about 128,000 tokens, and a single page alone can fill a 128k context window.
Context Lens shows a comparison of raw tokens versus mapped tokens for each repository and validates the results against a real o200k tokenizer run.
The o200k_base vocabulary is expanded to approximately 200,000 tokens, offering more efficient encoding of common symbols and multi-byte characters in code compared to cl100k_base.
All Facts (9)
The GPT-4 series uses the cl100k_base vocabulary (approximately 100,000 tokens), while the GPT-4o series upgrades to o200k_base (approximately 200,000 tokens).
90%UnverifiedThe authors reproduced the previously published "The Tokenizer Tax" table, with all 66 of 66 cells within 0.5% error; the average tax rate is 8.03x under cl100k_base (8.0x in the original paper) and 2.14x under o200k_base (2.1x in the original paper).
50%UnverifiedThe author's measurements show that the JSON Schema definitions of 18 tools consume 2,388 tokens, averaging 133 tokens per tool, and in a 10-step Agent run they can account for 36% of input tokens and 32% of the total bill.
50%Unverifiedo200k_base is the tokenization encoding used by newer GPT series, with a larger vocabulary, and its segmentation efficiency for structured text such as code and JSON is usually better than the older cl100k_base.
50%UnverifiedEncoded with o200k_base, 'I love strawberry' becomes [40, 3047, 101830].
50%Unverifiedo200k_base is the tokenizer OpenAI introduced for the GPT-4o series, with a vocabulary of about 200,000, and its compression efficiency for multilingual (especially non-English) text is clearly better than the earlier cl100k_base.
50%UnverifiedIn a measured sample of 20 publicly available German-language pages, the median page's raw HTML was about 128,000 tokens, and a single page alone can fill a 128k context window.
50%UnverifiedThe o200k_base vocabulary is expanded to approximately 200,000 tokens, offering more efficient encoding of common symbols and multi-byte characters in code compared to cl100k_base.
50%UnverifiedContext Lens shows a comparison of raw tokens versus mapped tokens for each repository and validates the results against a real o200k tokenizer run.
50%