Count Tokens Before You Paste: A Practical Guide to LLM Token Math

When you paste a long document into ChatGPT, Claude, or Gemini, three common failure modes tend to show up: the answer ignores the ending, the context limit gets triggered, or the API bill comes in higher than expected. All three trace back to the same number—tokens.

What Tokens Actually Are

Models don't read characters or words directly. A tokenizer slices text into chunks according to a fixed vocabulary, and all the model sees is a sequence of chunk IDs.

• Common English words are usually one token: the, house, token
• Longer or rarer words get split apart: tokenization → token + ization
• Spaces and punctuation count too, and leading spaces often belong to the next token
• Numbers, code, URLs, and non-Latin scripts consume more tokens per character

That last point is the most underestimated. The same sentence translated into Hindi, Japanese, or Arabic can take two to three times as many tokens as the English version; compressed JSON or a stack trace is far denser than ordinary prose. Understanding this difference is the first step in estimating cost.

Tokenizers and models are two separate things. The tokenizer is the "translation layer" between text and model—its vocabulary is fixed before the model is trained and never changes afterward. That means the same word can be split into completely different chunks across model families. GPT-4o's o200k_base vocabulary has roughly 200,000 entries and represents multilingual text more efficiently, while older models use cl100k_base with only about 100,000 entries and split rare words more finely. The tokenization algorithm itself is usually BPE (byte pair encoding) or a variant: starting from single characters, it repeatedly merges the most frequent adjacent pairs until the vocabulary is full. The statistics come from the training corpus, so an English-dominant corpus makes English words split into more "whole" pieces while other languages get chopped more finely—this is the root cause of higher token density in non-English text, not any model "weakness" in a given language.

Mental Math Is Enough: Rules of Thumb

For English prose and OpenAI's current tokenizers:

• 1 token ≈ 4 characters
• 1 token ≈ ¾ of a word, i.e. 100 words ≈ 130 tokens
• A dense A4 page ≈ 600–800 tokens

Other vendors have published their own rough ratios with slightly different numbers, because each model family uses a different tokenizer. Treat these as estimates—when a decision really matters, count for real.

Counting Precisely with Code

OpenAI's tokenizer is open source. In JavaScript, js-tiktoken runs in any environment, including the browser:

import { Tiktoken } from "js-tiktoken/lite";
import o200k_base from "js-tiktoken/ranks/o200k_base";

const enc = new Tiktoken(o200k_base);
const text = "Count your tokens before you paste.";
console.log(enc.encode(text).length); // Token count

o200k_base is the encoding used by GPT-4o and newer OpenAI models, while older GPT-4 and GPT-3.5 use cl100k_base. In Python the equivalent is the tiktoken package:

import tiktoken

enc = tiktoken.get_encoding("o200k_base")
print(len(enc.encode("Count your tokens before you paste.")))

Claude and Gemini don't publish their tokenizers, but both APIs offer counting endpoints (Anthropic's count_tokens, Gemini's countTokens) that you can use when you need an exact number before sending a request.

Note that encode returns an array of token IDs, and its length is the token count—but it does not equal the total tokens the model actually processes. Most APIs automatically append chat template markers (such as special role delimiters) before and after the input, and this "hidden cost" typically runs into the tens of tokens. It's negligible for short prompts, but across batch calls or very short requests it accumulates into a noticeable deviation. The count also depends on which vocabulary you use: estimating a GPT-4o request with cl100k_base will systematically overshoot. So the correct approach to accurate pricing is: first confirm the encoding name that matches the target model, then count, and finally add the fixed overhead of the chat template. Since Claude and Gemini's tokenizers aren't public, you can only rely on their official counting endpoints—their results can't be reproduced locally.

What Does a Prompt Actually Cost?

All API price lists are quoted per million tokens, with input and output billed separately:

cost = input_tokens  / 1,000,000 × input_price
     + output_tokens / 1,000,000 × output_price

Walk through it with a hypothetical round number: assume input costs $2 per million tokens and output costs $8. A 3,000-token prompt plus a 500-token answer costs:

3,000 / 1,000,000 × $2 = $0.006
  500 / 1,000,000 × $8 = $0.004
                 total = $0.010 per request

One cent looks trivial—until it runs 50,000 times a day. Two frequently overlooked points:

1. Chat history is input every single time. In a long conversation, every new message resends the entire thread, and input tokens grow with each turn.
2. Output is usually the more expensive side. Asking for "brief" answers or a fixed format often saves more than trimming the prompt.

The context window and billing are two related but distinct things. The context window is the maximum total tokens the model can "see" in one request, including the system prompt, chat history, current input, and the output about to be generated—ranging from 8K to 200K across models. Billing, meanwhile, is calculated separately on actual input and output tokens consumed. The key point: even if chat history is long "stale," as long as it's still in the window, it's re-billed as input on every turn. In other words, a 20-turn conversation means turn 20 pays for all the content of the previous 19 turns. This is why the marginal cost of a long session rises with each turn rather than staying constant. When history approaches the window limit, some APIs trigger truncation or errors—which is the shared mechanism behind the "ignores the ending" and "hits the limit" failure modes from the opening.

Five Ways to Save Tokens Without Losing Quality

1. Strip HTML and template content first: delete navigation, footers, and tags before pasting a webpage—they're pure waste.
2. Compress or trim JSON: send only the fields the model needs, not the entire API response.
3. Summarize old context: replace the first 20 messages of a long thread with a five-line summary.
4. Don't repeat instructions in every message: put them in the system prompt once.
5. Check non-English and code-heavy input: these two categories most easily push token counts past your expectations.

A Quick Check Without Writing Code

If you just want to paste some text and see the count, the author offers a free token counter that runs entirely in the browser—your pasted content is never uploaded. It gives exact counts for OpenAI tokenizers, labels Claude and Gemini results as estimates, and includes a cost calculator with a dated, editable price table.

What's the biggest token surprise you've run into: an overlong chat history, a non-English document, or a JSON blob that ate your entire context window?

One-line Summary

Tokens are the basic unit of text that models process—understanding how to count, estimate, and cost them prevents context overflows and runaway bills.

Paragraph Summary

Starting from three common failure modes when pasting long documents into large models, the article shows they all stem from the basic unit of tokens. Tokenizers slice text by a fixed vocabulary; common English words are usually one token, while rare words, non-Latin scripts, code, and URLs have significantly higher token density. The article gives rules of thumb for English prose (1 token ≈ 4 characters ≈ ¾ word, 100 words ≈ 130 tokens) and demonstrates precise counting with js-tiktoken and Python's tiktoken, plus counting endpoints from Claude and Gemini APIs. On cost, input and output are billed separately per million tokens, and chat history being resent every turn plus output being pricier are two easily overlooked expense sources. It closes with five token-saving strategies and a browser-based counter that requires no code.