ttok 0.4 Released: Token Counting CLI Tool Adds --list-models Command

After Two Years, ttok Gets a Major Update

Developer Simon Willison has released version 0.4 of ttok. Built on OpenAI's open-source tiktoken library, this command-line tool focuses on solving one specific yet frequently encountered problem: counting the number of tokens in a text. Although the core functionality remains unchanged, this update fixes long-standing Click warnings, refreshes the CI pipeline, and adds a --list-models command for listing currently supported models.

For developers who work with LLMs on a daily basis, token counting is hardly a trivial detail. API call costs, context window limits, and truncation strategies in prompt engineering all depend directly on accurate token estimates. ttok's value lies in packaging these capabilities into a single simple pipe command.

Zero-Install Usage with uvx

One notable detail in this update is ttok's support for uvx. uvx comes from the Python package management tool uv and lets users run CLI tools directly without explicitly installing them. This means you can count tokens in any file like this:

cat file.txt | uvx ttok

This design lowers the barrier to entry. For users who only occasionally need to count tokens, there's no need to configure Python packages in the local environment, and version conflicts are avoided. It turns ttok from "a tool you need to install" into "a command you can use on the spot" — a substantial usability improvement for a single-purpose CLI tool.

The relationship between uvx and uv: uvx is a convenient alias provided by uv, effectively equivalent to uv tool run. uv is a high-performance Python package and project manager written in Rust by Astral (the developers of Ruff), aiming to fully replace traditional tools like pip, pip-tools, virtualenv, and pipx in terms of speed. When you run uvx ttok, uv first resolves the latest version of ttok, installs it into an isolated cache environment managed uniformly by uv, and then runs the entry script. The whole process doesn't pollute the current project's virtual environment, nor does it require you to manually run pip install. The first run incurs a download cost, but subsequent runs hit the cache and are nearly instantaneous.

This mechanism is similar to pipx, but uv relies on its global cache and hard-linking strategy to allow multiple tools to share dependencies, avoiding redundant downloads. For small tools like ttok that are single-purpose and frequently invoked, uvx offers a best practice somewhere between "global installation" and "throwaway script": it guarantees version isolation while remaining instantly available.

The Significance of the New --list-models Command

The --list-models command added in version 0.4 lets users directly view the models currently supported by tiktoken and their corresponding encodings.

This improvement addresses a real pain point: different models (such as GPT-4, GPT-3.5, text-embedding-ada-002) use different tokenizers, and the same text may have a different token count under each model. Previously, users had to consult documentation or source code to confirm the encoding name for a given model; now they can get it directly from the command line.

uvx ttok --list-models

For teams that need precise cost control, this command reduces the risk of count discrepancies caused by using the wrong encoding.

The difference between tokenizer and encoding: Here it's worth distinguishing two easily confused concepts. A tokenizer refers to the complete algorithmic pipeline that splits text into tokens (including preprocessing, BPE merge rules, etc.), while an encoding is the name OpenAI gives in tiktoken to each specific set of tokenizer rules, such as cl100k_base, o200k_base, p50k_base. The same encoding may be shared by multiple models — for example, early versions of GPT-4 and text-embedding-ada-002 both use cl100k_base.

So --list-models lists "model name → encoding name" mappings. tiktoken itself does not include a complete list of all models (it only ships encodings); ttok's role is to associate common model aliases with their corresponding encodings so users don't have to remember whether gpt-4o should use o200k_base or cl100k_base. This also explains why model list updates often lag behind OpenAI's release of new models — the mapping table requires manual maintenance.

Maintenance Status and Tool Positioning

In the release notes, Simon Willison openly acknowledges that ttok has gone "two years without an update." This candor itself reflects the norm for small open-source tools: when the core dependency (tiktoken) is stable and functional requirements haven't changed, the tool enters a low-maintenance period. This update is more about clearing technical debt — fixing Click library deprecation warnings, updating CI configuration, and ensuring the project still works properly in modern Python environments.

ttok's positioning has always been clear: a focused, single-responsibility CLI tool. It doesn't try to become a full tokenizer management platform, nor does it offer a graphical interface — it just quietly gets the counting job done. In today's increasingly complex AI toolchain, this restrained design has actually helped it stay relevant.

For developers who frequently need to estimate token usage, version 0.4 is worth upgrading to; and for those who haven't tried it yet, the one-line uvx invocation may be the lowest-cost way to get started.

One-line Summary

ttok 0.4 clears technical debt and adds --list-models, with uvx zero-install support.

Paragraph Summary

Simon Willison has released ttok 0.4, the first update in two years to this command-line tool built on OpenAI's tiktoken library that counts the number of tokens in a text. The core of this update is clearing technical debt: fixing Click library deprecation warnings, updating CI configuration, and adding a --list-models command that lets users directly view each model's corresponding encoding, avoiding count discrepancies caused by picking the wrong tokenizer. The article particularly highlights ttok's support for uvx — leveraging uv's isolated cache mechanism, users can run a pipe command like cat file.txt | uvx ttok to count tokens with zero installation. The author argues that, against the backdrop of an increasingly complex AI toolchain, a single-responsibility, restrained CLI tool like ttok has actually retained lasting vitality.