ttok 1.0 Released: Default Tokenizer Switches to GPT-5/GPT-6

Simon Willison's token counting tool ttok has officially reached version 1.0. The change in this release looks minor—just switching the default tokenizer from GPT-4 to GPT-5/GPT-6—but it exposes a broader synchronization problem between developer tools and model iteration.

Starting with a Default Value

After releasing ttok 0.4, Simon Willison ran uv tool upgrade ttok to upgrade the tool, then piped a file into the new version. He found that the tool still used the GPT-4 tokenizer by default.

With the GPT-5 series now widespread and the GPT-6 series beginning to appear, this default was clearly outdated. Simon decided to take the opportunity to release 1.0 directly, rather than continue patching on the 0.x line.

For developers who routinely estimate API call costs or verify prompt lengths, whether a token counting tool's default tokenizer matches the model actually being called directly affects estimation accuracy.

A tokenizer is a low-level component that large language models use to process text, responsible for splitting raw strings into token sequences the model can recognize and mapping them to integer IDs. The same piece of text may be split into different numbers of tokens under different tokenizers, and API billing and context window limits are both measured in token counts. The tokenizer therefore determines judgments like "how much will this prompt cost" and "will it exceed the limit." The GPT-4 and GPT-5 series do not use entirely identical tokenization schemes, so if a tool defaults to an old tokenizer to estimate token counts for a new model, the results may show systematic bias.

tiktoken is OpenAI's open-source token counting library, and tools like ttok typically call it directly to ensure counts match the official API. When OpenAI releases new models without publishing their tokenizer details in sync, tiktoken becomes the community's main source for tracking tokenization changes.

The Mystery of GPT-6's Tokenizer

The problem is that OpenAI has not formally confirmed whether GPT-6 uses the same tokenizer as the GPT-5 series. There is an issue in the tiktoken repository about this, with fairly heated discussion, but the official side has never given a clear statement.

This means that ttok switching its default to the GPT-5/GPT-6 tokenizer is, in a sense, based on inference rather than official documentation.

Simon cited an experiment by William Liu as the basis. In the tiktoken repository discussion, William Liu compared the tokenization results of multiple GPT models using a set of fixtures.

Fixtures refer to a set of fixed input samples and their expected outputs in software testing, used to make repeatable comparisons between different implementations or versions. For tokenizer questions, fixtures are usually several carefully selected passages of text—covering English, code, multiple languages, special symbols, and so on—with each model's reported token count for them recorded. As long as the input stays the same, anyone rerunning it gets the same result, so fixtures are often used as evidence of cross-model behavioral consistency. However, fixtures can only prove consistency "on these samples" and cannot exhaust all input distributions, which is the fundamental reason they cannot replace official statements.

What the Experimental Data Shows

William Liu's experiment covered seven GPT models, including three variants of GPT-5.5 and GPT-5.6 (Sol/Terra/Luna), as well as three variants of GPT-6 (Astra/Sol/Luna).

Across all 31 test samples, these seven models reported exactly the same token count: 44,794.

This result indicates that, at least within the corpus William Liu tested, GPT-6 introduced no change in input counting. This provides a verifiable reference for ttok's switch of its default tokenizer, although it cannot replace official confirmation.

Practical Impact on Developers

As a command-line tool, ttok is mainly used to quickly count tokens in text from the terminal. Developers can use it to estimate API call costs and check whether a prompt exceeds the context window.

After the default tokenizer switches to GPT-5/GPT-6, developers using the new models no longer need to manually specify a tokenizer, and the tool's default behavior is more consistent with actual usage scenarios.

But there is also a potential risk here: if OpenAI later adjusts the tokenizer in the official GPT-6 release, defaults based on current experimental data may deviate. Developers still need to watch for subsequent updates in the tiktoken repository.

For scenarios requiring precise calculation, it is advisable to explicitly specify the tokenizer version in code rather than relying on the tool's default.

The context window refers to the maximum total number of tokens a model can process in a single request, including the input prompt and the model output. Once exceeded, the API will directly return an error or truncate the content. Because different models may have different window sizes and tokenization methods, developers who use the wrong tokenizer to estimate length may misjudge near the limit—thinking there is still room when the limit has actually been exceeded. Explicitly specifying the tokenizer version and doing boundary tests on critical paths are common ways to avoid this kind of problem.

The Pacing Problem of Tool Iteration

The release of ttok 1.0 reflects a common phenomenon: tools built around large models often iterate more slowly than the models themselves update.

Tokenizers are a low-level component of model behavior, but their rate of change and degree of public disclosure are not high. Tool authors need to fill information gaps through community experiments when official documentation is lacking.

Simon Willison's choice to use this default change as the reason for a 1.0 release is also a signal: the tool is stable enough to enter an official version. Future maintenance priorities may shift toward supporting more models and following upstream tiktoken updates.