Counting LLM Tokens in Python: tiktoken vs. Zero-Dependency Heuristics

Every LLM feature ultimately has to answer one question: how many tokens is this text? Context windows are billed by token, costs are calculated by token, and truncation happens by token. Character counts and word counts may look close, but they are neither reliable nor accurate.

This article introduces two counting methods and shows how to elegantly hide the choice between them behind a unified interface.

Method 1: Exact counting with tiktoken

tiktoken is the byte pair encoding (BPE) tokenizer used by several OpenAI models. It gives the token count the model actually sees. If billing, hard context limits, or precise prompt trimming are involved, you must use exact counting.

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")
tokens = enc.encode("Counting tokens is deceptively subtle.")
print(len(tokens))  # exact token count

There are two key points here. First, the encoding matters. cl100k_base corresponds to one family of models; other models use different vocabularies, so you need to choose the encoding your target model actually uses. Second, encode raises an error when handling text containing special tokens unless you explicitly tell it not to treat them specially. In production, I always pass disallowed_special=() so that arbitrary user text cannot trigger an exception.

def count(text: str) -> int:
    if not text:
        return 0
    return len(enc.encode(text, disallowed_special=()))

The cost of exactness is a dependency: tiktoken is a compiled package that downloads vocabulary files, and it is not always suitable for lightweight libraries, offline CI environments, or restricted environments.

BPE (Byte Pair Encoding) is the mainstream algorithm for modern LLM tokenization. Its core idea is to start from individual bytes or characters, repeatedly count the most frequent adjacent symbol pairs in the corpus, merge them into new symbols, and iterate until the vocabulary reaches a preset size. In the resulting vocabulary, common words (such as the and is) often occupy a single token, while rare words are split into multiple subword fragments.

The key point is that BPE operates at the byte level, so any Unicode text (including emoji, Chinese, and code symbols) can be encoded losslessly without characters that cannot be represented. This also explains why the same word may have different token counts across models: each model is trained on different corpora and vocabulary sizes, so the merge rules naturally differ. cl100k_base is the vocabulary trained by OpenAI for the GPT-3.5/GPT-4 series, with around 100,000 merge rules, so different models must match their respective encoding names and cannot be mixed.

Method 2: Zero-dependency heuristic estimation

Sometimes you do not need an exact number. When I am analyzing which part of a prompt consumes the most context, or observing the growth trend of usage in a conversation, relative size matters more than an exact integer. In that case, use a pure-Python heuristic: zero dependencies, no network required.

The idea is that real BPE tokenizers roughly pack several characters into one token and split long words into multiple fragments. So split the text into words, numbers, and symbol segments, and charge more tokens for longer segments.

import re


The `[^\W\d_]+` mentioned in the text uses Python regex's negated character class shorthand. Understanding its meaning helps explain why the heuristic is designed this way:

- `\W` means non-word characters (that is, not letters, digits, or underscores)
- `\d` means digits
- `_` is an underscore
- Therefore `[^\W\d_]` is equivalent to "is a word character, but not a digit, and not an underscore," that is, a **Unicode letter**

With the `re.UNICODE` flag, `\w` matches letters from all languages (Chinese, Arabic, Cyrillic, and so on), not just ASCII. This is exactly why the heuristic can roughly handle multilingual text. `\d+` captures consecutive digits, and `[^\s\w]` captures a single non-space symbol (such as punctuation or brackets). The three alternatives match in alternation, splitting text into "letter blocks, digit blocks, and symbol units," then estimating at roughly 1 token per 4 characters. This 4-characters ratio roughly corresponds to the average compression rate of English BPE, but the deviation grows noticeably in Chinese, code, and similar scenarios.
# Words (including Unicode letters), numbers, or any single non-space symbol
_TOKEN_RE = re.compile(r"[^\W\d_]+|\d+|[^\s\w]", re.UNICODE)
_CHARS_PER_PIECE = 4


def heuristic_count(text: str) -> int:
    if not text:
        return 0
    total = 0
    for match in _TOKEN_RE.finditer(text):
        piece = match.group(0)
        n = len(piece)
        if n <= _CHARS_PER_PIECE:
            total += 1
        else:
            # Split long fragments into subtokens, rounding up
            total += (n + _CHARS_PER_PIECE - 1) // _CHARS_PER_PIECE
        # Whitespace is folded into adjacent tokens and not counted separately
    return total

This code runs in any Python environment, and its deterministic output lets tests and CI run reliably without heavyweight installation.

Honest caveat: this is entirely approximate. It does not reproduce any real BPE vocabulary, and its results will differ from the model's actual count, potentially significantly on code, non-English text, or symbol-heavy input. Never use a heuristic to bill customers or enforce hard context limits. It is suitable for relative comparisons and offline analysis; scenarios with consequences must use tiktoken.

Unified interface: no more choosing between two

The real trick is not choosing which counter to use, but making the rest of the code not care which one is used. Define an abstract interface that both implementations satisfy:

from abc import ABC, abstractmethod


class Tokenizer(ABC):
    name: str = "base"

    @abstractmethod
    def count(self, text: str) -> int:
        raise NotImplementedError

    def count_all(self, texts: list[str]) -> list[int]:
        return [self.count(t) for t in texts]

Both counters are subclasses. The heuristic version is always available. The tiktoken version imports lazily inside __init__, so it only fails when actually constructed without the dependency present:

class TiktokenTokenizer(Tokenizer):
    def __init__(self, encoding: str = "cl100k_base") -> None:
        try:
            import tiktoken
        except ImportError as exc:
            raise ImportError(
                "tiktoken is not installed; install it to get exact counts"
            ) from exc
        self._enc = tiktoken.get_encoding(encoding)
        self.name = f"tiktoken:{encoding}"

    def count(self, text: str) -> int:
        if not text:
            return 0
        return len(self._enc.encode(text, disallowed_special=()))

Note that the import is inside the constructor, not at the top of the module. This is the key to graceful degradation: importing the module itself does not require the dependency, and the ImportError only appears when the caller requests functionality that cannot be satisfied.

A factory function combines the two. auto prefers exact and silently falls back to heuristic:

def _try_tiktoken(encoding: str = "cl100k_base"):
    try:
        return TiktokenTokenizer(encoding)
    except ImportError:
        return None


def get_tokenizer(name: str = "auto") -> Tokenizer:
    name = (name or "auto").lower()
    if name == "heuristic":
        return HeuristicTokenizer()
    if name == "tiktoken":
        tok = _try_tiktoken()
        if tok is None:
            raise ImportError("tiktoken requested but not installed")
        return tok
    if name == "auto":
        return _try_tiktoken() or HeuristicTokenizer()
    raise ValueError(f"unknown tokenizer: {name!r}")

The design choices are deliberate. auto gives the best answer currently available without crashing; heuristic guarantees a dependency-free path, and tests should pin it to ensure they do not depend on optional installations; tiktoken is explicit and loud: if exactness is requested but cannot be satisfied, it errors out rather than silently approximating. This last distinction matters because silently replacing exact with approximate is exactly the bug that leads to unexpected overage bills.

Lazy import is a pattern that moves an import statement from the top level of a module into a function or method. Python executes top-level code the first time a module is imported. If import tiktoken is at the top, then any import this_module operation will immediately try to load tiktoken and crash if the dependency is missing.

By putting the import inside __init__, the module itself can load normally in an environment without tiktoken, and the dependency check is only triggered when a TiktokenTokenizer instance is actually constructed. This is the premise that makes the "graceful degradation" described in the article possible. The tradeoff is that each instance construction performs an import lookup (Python has a module cache, so the overhead is very small), and import errors can only surface at runtime rather than startup. This tradeoff is worth it for optional dependencies, but for core dependencies it is still better to keep them at the top level so failures happen early.

Benefits

The rest of the application no longer touches tiktoken or regex. It just calls tokenizer.count(text). Switching counters requires changing one line at the edge; tests run the heuristic version offline, and production gets exact counts wherever the dependency is available. Keeping the name field lets reports explain where the number came from, because "1,240 tokens (heuristic)" and "1,240 tokens (tiktoken)" mean different things to readers.

The whole pattern is: abstract count, lazy optional dependencies, and an auto mode that prefers exact and degrades gracefully. If you want to see how this plugs into a real tool for analyzing the distribution of tokens in a prompt, it comes directly from my project ctxlens, and the full tokenizer package is at github.com/AgentPostmortem/ctxlens. Borrow the interface, keep the caveat, count with confidence.

One-line Summary

Unify exact tiktoken counting and zero-dependency heuristic estimation behind an abstract interface, with an auto mode that prefers exact and degrades gracefully.

Paragraph Summary

The article points out that every LLM feature cannot avoid the token counting problem, and that character counts and word counts are both unreliable. The author proposes two approaches: first, exact BPE counting with tiktoken, suitable for billing, hard context limits, and precise trimming, but dependent on a compiled package and vocabulary downloads; second, pure-Python regex heuristic estimation, zero-dependency and deterministic, suitable for offline analysis and relative comparison, but explicitly not for billing or hard limits. The real trick is to define an abstract Tokenizer interface so the rest of the code does not care about the concrete implementation, using lazy imports and a factory function to implement an auto mode that prefers exact and silently falls back to heuristic when the dependency is missing, while retaining explicit heuristic and tiktoken modes to distinguish testing from production needs.