Why Can't LLMs Count the R's in Strawberry? A Deep Dive into Tokenizer Mechanics

Ask a language model how many r's are in "strawberry" and, for a long time, it will answer "two." The same model can write a sonnet about strawberries, explain their genetic makeup, and translate the word into forty languages. So why does it stumble on counting letters?

The answer lies in the tokenizer. Before a prompt reaches the first Transformer layer, a separate program has already sliced the text into chunks, each swapped for an integer. The model reads only these integers from start to finish. If "strawberry" becomes a single integer, the three r's are hidden inside a number the model cannot pry apart.

Tokenization is the least glamorous part of the LLM stack, yet it causes a surprising number of problems: unstable multi-digit arithmetic, higher costs for non-English users, "glitch tokens" that make models behave oddly, and token counts that never match your character-based estimates. This article, based on actual measurements with OpenAI's open-source tiktoken library on the GPT-2, GPT-4 (cl100k_base), and GPT-4o (o200k_base) vocabularies, breaks down each of these issues one by one.

Tokens: The Only Alphabet the Model Knows

Neural networks cannot do string operations; they need numbers. The most naive approach is to give every character an ID, but a two-thousand-word document would become over ten thousand positions, and every Transformer layer would pay a computational cost for each one. The other extreme is to give every word an ID, but no fixed list can cover all the words, names, typos, URLs, and code identifiers people will type.

Rico Sennrich and colleagues quantified this trade-off in their 2016 paper "Neural Machine Translation of Rare Words with Subword Units." On a roughly 100-million-word German training corpus, keeping whole words required a vocabulary of 1.75 million entries, yet still left 1,079 unknown words in the test set. Splitting into characters required only about 3,000 symbols but inflated the corpus to 550 million positions. Their proposed subword method, BPE, landed in the middle: about 63,000 symbols, 112 million positions, and zero unknown words.

That middle product is the token: usually shorter than a word for rare words, and exactly one word for common ones. Modern LLM vocabularies range from 50,000 to 200,000 entries. tiktoken's README gives an average of about 4 bytes of text per token. Measured on English text, the GPT-4o tokenizer averages about 4.4 characters per token, or roughly 0.63 words.

BPE (Byte Pair Encoding) is a core concept that recurs throughout this article. It originated as a data compression algorithm proposed by Philip Gage in 1994: repeatedly find the most common adjacent byte pair in the data and replace it with a new unused symbol, thereby shortening the representation. Rico Sennrich and colleagues adapted this mechanism in 2016 into a method for building subword vocabularies—splitting every word in a corpus into characters, repeatedly merging the most frequent adjacent symbol pairs, with each merge producing a new token and recording a rule. It is used because it strikes a balance between the two extremes of "whole word" and "single character": common words collapse into a single token, rare words split into several subword fragments, controlling vocabulary size while producing almost no unrepresentable input.

The Five Steps from Text to Token ID

Take the sentence "I love strawberry" entering GPT-4o as an example:

Step one, raw text. The string arrives as UTF-8 bytes. Some tokenizers first normalize (Unicode normalization, lowercasing in older models), but the GPT series keeps bytes as-is.

Step two, pre-tokenization. A regular expression splits the text into fragments that merge rules can never cross: letter runs, number runs, punctuation, whitespace. The GPT-2 version attaches leading spaces to the following word, so the fragments are I, love, strawberry. This means " strawberry" with a space and "strawberry" without one are two different tokens.

Step three, BPE merges. Each fragment starts as a byte sequence, and the tokenizer applies merge rules in learned order, gluing adjacent pairs together until no rule applies. Common words collapse into a single token; rare words become several subword chunks.

Step four, token IDs. Each final chunk is looked up in the vocabulary. Encoded with o200k_base, "I love strawberry" becomes [40, 3047, 101830]: three integers.

Step five, embeddings. Inside the model, each ID selects a row of the embedding matrix—a vector of several thousand dimensions. From then on the Transformer deals only with these vectors.

This design yields two properties: tokenization is lossless, since decoding the IDs restores the original bytes; and tokenization works for any text, including strings never seen during training. The latter relies on the byte-level BPE trick.

How BPE Learns Its Own Vocabulary

Byte Pair Encoding was originally a data compression algorithm (Philip Gage, 1994): repeatedly find the most common adjacent byte pair and replace it with a new unused symbol. Sennrich and colleagues turned this idea into a method for building vocabularies. The training process is as follows:

1. Split every word in the training corpus into characters; the initial vocabulary is these characters.
2. Count the occurrences of every adjacent symbol pair, weighted by word frequency.
3. Merge the most frequent pair into a new symbol, add it to the vocabulary, and record the rule.
4. Repeat until the vocabulary reaches the target size.

The number of merges is the only parameter that truly needs setting. The final vocabulary size equals the number of starting symbols plus the number of merge operations.

Take the toy corpus from the Hugging Face LLM course: "hug" (10 times), "pug" (5 times), "pun" (12 times), "bun" (4 times), "hugs" (5 times). The initial vocabulary is b g h n p s u. First merge: u + g appears 20 times, higher than any other pair, so ug becomes a token. Second merge: u + n leads with 16, so un joins. Third merge: h + ug appears 15 times, and the first three-letter token hug is born.

Real tokenizers run this loop tens of thousands of times over gigabytes of text. GPT-2's vocabulary has 50,257 entries. GPT-4's cl100k_base has about 100,000, and GPT-4o's o200k_base about 200,000. Meta says Llama 3 moved to a 128K token vocabulary, using up to 15% fewer tokens than Llama 2 on the same text.

Encoding new text means replaying the merge rules: split into characters (or bytes) and apply the learned rules in priority order. Words common in training collapse into a single token; unseen words can still be encoded, just in smaller fragments. With the GPT-4o vocabulary, "unbelievably" becomes un | bel | ievably.

Merge order also explains some odd phenomena. Merge rules reflect the tokenizer's training data, not necessarily the model's. This point resurfaces in the glitch token section.

Byte-Level BPE: Saying Goodbye to "Unknown Words"

Classic BPE starts from characters, and Unicode defines about 150,000 of them. Any character missing from the training corpus becomes [UNK], the "unknown" token. This is one reason early NLP models handled emoji poorly.

GPT-2 introduced a simple fix: run BPE on bytes instead of characters. There are only 256 byte values, the base vocabulary is fixed at 256 entries, and any input—whether emoji, Tamil, or binary garbage—can be written with them. Merge rules then progressively build up common multi-byte sequences. GPT-2 effectively has no unknown tokens, because with byte-level BPE an unknown character cannot occur.

The byte fallback is visible firsthand: in the GPT-2 vocabulary, the emoji 🙂 (4 UTF-8 bytes) becomes 2 tokens; GPT-4o's larger vocabulary learns the whole sequence, so it is 1 token. The cost is that rare writing systems may be sliced into fragments that are not even complete characters, which leads to the "multilingual token tax" below.

UTF-8 is a variable-length character encoding: ASCII characters take 1 byte, many European characters take 2, CJK characters usually take 3, and emoji generally take 4. Byte-level BPE starts from a fixed set of 256 byte values precisely because UTF-8 guarantees that any Unicode text can be represented as a byte sequence. This also explains the phenomenon mentioned in this article: a 4-byte emoji gets sliced into multiple tokens under a small vocabulary, while a large vocabulary, having learned the full sequence through merge rules, needs only one token. The cost is that when the byte combination of a writing system is too rare in the training corpus, the tokenizer may cut fragments that cross character boundaries, not even forming a complete visible character.

Why the Model Cannot See Those r's

Back to the strawberry problem. The GPT-4o tokenizer processes the classic question as follows: "How many r's are in strawberry?" becomes 8 tokens: How | ·many | ·r | 's | ·are | ·in | ·strawberry | ?. The word you care about is token #101830. To the model, this is an opaque symbol mapping to an embedding vector. There is no "r" inside to count.

Any knowledge the model has about the spelling of strawberry must be memorized from places in the training text where the spelling happened to appear—dictionaries, spelling bee pages, children's workbooks. This knowledge is thin and indirect, so the model can only guess, and for a word ending in a double letter, "two" is a reasonable guess.

The segmentation also depends on context in ways people do not expect: strawberry with a leading space (common mid-sentence) is 1 token; strawberry without a space (say, at the start of a message) is 3 tokens: st | raw | berry; capitalized Strawberry is another 3: Str | aw | berry. The model may have learned slightly different things about the same word.

The fix is always to expose the letters. Writing the word as s-t-r-a-w-b-e-r-r-y produces 10 tokens, one per letter, at which point counting becomes something the model can actually do. Better yet: when an exact character-level answer matters (counting, reversing strings, checking regexes), have the model call a code tool. One line of Python: "strawberry".count("r") always returns 3.

The Multilingual Token Tax

A tokenizer's merge rules come from its training data, and that data skews heavily toward English and code. English words get dedicated tokens; other languages—especially those with their own writing systems—get fewer merges and stay closer to raw bytes.

The short sentence "The weather is nice today" and its simple translations, under OpenAI's three tokenizers: English is 6 tokens under all three. Tamil is 85 tokens under the GPT-2 tokenizer, 42 under GPT-4, and 11 under GPT-4o. Hindi goes from 27 to 21 to 5, and Russian from 27 to 13 to 6.

Aleksandar Petrov and colleagues conducted a systematic study in their 2023 paper "Language Model Tokenizers Introduce Unfairness Between Languages." They found that the token length difference for the same text across languages can reach 15 times, and even byte-level and character-level models show gaps of more than 4 times. The consequences are concrete: APIs bill by token; a 128K-token context window holds far fewer pages of Tamil than of English; output is generated token by token, affecting latency. If you serve non-English markets, measure token counts on real traffic, and do not assume "about 4 characters per token."

Numbers: Why Long Addition Goes Wrong

The same pre-tokenization regex decides how numbers are chunked, and choices vary widely across labs. Aaditya Singh and DJ Strouse laid this out in their 2024 paper "Tokenization counts":

• Pure BPE (GPT-3, Claude 2.1, OLMo): whichever digit string happens to be frequent gets a token. 710 might be one token while 711 is not, so same-length numbers are segmented inconsistently.
• Single digits (PaLM, Llama 1 and 2, Mistral): each digit becomes its own token.
• Up to 3 digits, left to right (GPT-3.5, GPT-4): the \p{N}{1,3} rule in the tiktoken pattern. GPT-4o's o200k_base uses the same rule.

Left-to-right chunking means 1234567 becomes 123 | 456 | 7. The chunks align with the start of the number, while place value and carries in addition begin at the end. Add commas and the tokenizer splits at the commas, giving 1 | , | 234 | , | 567, with every chunk landing on a thousands boundary.

Singh and Strouse tested this with 7- to 9-digit addition on GPT-3.5. With plain digits, 8-shot accuracy was 75.6%; with commas, 97.8%. When the answer has more digits than the input (because of the final carry), plain-digit accuracy drops to 8.25%, and the errors are not random: the model is almost always right in the first three digits and wrong in the fourth, exactly at the first token boundary. Asking the model to first rewrite the problem with commas restores most of the lost accuracy. GPT-4 is less affected, but the effect does not disappear.

The takeaway is not "add commas everywhere," but that number formatting changes the problem the model has to solve. For scenarios requiring precision, such as invoices or financial calculations, hand the arithmetic to a calculator or code tool.

The regular expressions used in pre-tokenization (such as \p{N}{1,3} in the tiktoken pattern) scan the text before tokenization and carve out fragments that merge rules can never cross. \p{N} matches any digit character, {1,3} means match 1 to 3 of them, and the regex engine by default matches greedily from left to right, so long digit strings are cut into groups of three, aligned from the left. This is precisely the root of what the article calls "chunks aligned with the start of the number, while carries begin at the end": the token sequence the model sees is naturally grouped from most significant to least significant, whereas written addition needs to process digit by digit starting from the least significant place, and the two alignments disagree. Switching to comma separation happens to place every token boundary at a thousands grouping, matching the way humans write and do mental math, which is why accuracy rebounds sharply.

Glitch Tokens: When the Vocabulary Outlives the Data

In early 2023, Jessica Rumbelow and Matthew Watkins found a cluster of strange entries in the token embeddings of GPT-2 and GPT-3: SolidGoldMagikarp, TheNitromeFan, cloneembedreportprint, petertodd. In their LessWrong post, asking the model to simply repeat these strings produced evasion, insults, or substitutions. ChatGPT rendered " SolidGoldMagikarp" as "distribute." Some tokens even made GPT-3 give different answers at temperature 0—where it should be deterministic.

Part of their explanation goes back to merge rules: the tokenizer was trained on crawled text containing Reddit usernames, e-commerce site internal identifiers, and game logs, while the model was trained on more strictly filtered text. Strings like SolidGoldMagikarp (confirmed as a single token, #43453, in the GPT-2 vocabulary) got a vocabulary slot but almost never appeared in model training. Their embedding vectors stayed near their initial positions, and the model did not know what to do when it encountered them.

This is not just a GPT-2 curiosity. The 2024 paper "Fishing for Magikarp" built automatic detectors combining tokenizer analysis, weight signals, and prompt probing, and found undertrained tokens across many models. In the new OpenAI vocabulary, " SolidGoldMagikarp" is 5 ordinary tokens (·Solid | Gold | Mag | ik | arp), but every new tokenizer can bring its own oddities. If you train or fine-tune models, scanning for undertrained tokens is cheap insurance.

Other Tokenizer Families

BPE dominates today's LLMs, but you will also encounter two other algorithms:

WordPiece (developed by Google for BERT) also learns merges bottom-up, but it scores pairs as frequency divided by the product of the parts' frequencies, favoring pairs whose components are themselves rare. It keeps only the final vocabulary, and during encoding takes the longest known fragment for each word, then the next. In-word fragments carry a ## prefix, as in token ##ization.

Unigram LM comes from Taku Kudo's 2018 "Subword Regularization" and takes a top-down approach. It starts from a large candidate vocabulary, evaluates each fragment's contribution to a probabilistic model of the corpus, and prunes the least useful fragments. The same word can have multiple segmentations, and Kudo used sampled segmentations as noise to train translation models, making them more robust, especially in low-resource and out-of-domain settings.

SentencePiece (Kudo and Richardson, 2018) is a library rather than an algorithm, implementing both BPE and Unigram and training directly on raw sentences without language-specific pre-tokenization. Whitespace becomes a visible symbol ▁, so I saw a girl becomes ▁I ▁saw ▁a ▁girl. This makes it naturally suited to languages like Japanese and Chinese that do not put spaces between words.

The Vocabulary Size Trade-off

Why not set a million tokens? Each entry occupies a row in both the input embedding matrix and the output layer. At 4096 dimensions, a 200,000-entry vocabulary is about 820 million parameters per matrix. The more entries there are, the fewer training samples each rare token gets, which brings back the glitch token problem.

A smaller vocabulary means longer sequences: more attention overhead, faster fill-up of the context window, and more generation steps. Recent models trend toward larger vocabularies (50K, 100K, 128K, 200K), because the embedding table's share of total parameters keeps shrinking. Meta argues that Llama 3's 128K tokenizer delivered "significant model performance improvements."

The embedding matrix is the lookup structure that turns a token ID into a vector, with shape "vocabulary size × hidden dimension"; the output layer is usually a linear projection of the opposite shape, mapping the final hidden state back to scores for each token. The parameter counts in both places are proportional to vocabulary size, so doubling the vocabulary doubles both matrices at once. But as a share of total parameters, the embedding layer shrinks as models grow: a 200K-vocabulary, 4096-dimensional embedding matrix is about 820 million parameters, a heavy burden for a model of a few billion parameters but a rounding error for one of hundreds of billions. This is precisely the arithmetic reason recent models dare to expand vocabularies from 50,000 to 200,000—the gain from the higher average information content per token outweighs the marginal cost of the added parameters.

A Practical Guide for Engineers

Count with the real tokenizer. Use tiktoken.encoding_for_model("gpt-4o") for OpenAI models, or a provider's token counting endpoint, and measure before enforcing context limits or estimating costs. Character-based estimates go wrong for code, non-English text, and JSON.

Watch the boundary at the end of prompts. GPT-series tokenizers attach spaces to the following word, so " Paris" is a single token. A prompt ending with a trailing space ("The capital of France is ") leaves a bare space token, and the model must continue with "Paris" without its usual leading space, an unusual sequence rarely seen in training. Completion-style prompts should trim trailing whitespace.

Special tokens are control characters. Strings like <|endoftext|> map to reserved IDs (100257 in cl100k_base, 199999 in o200k_base) marking document and message boundaries. tiktoken refuses to encode them from ordinary text by default; keep that protection on, and pasted text cannot forge document or message boundaries.

Key Takeaways

• LLMs read integers, not letters. The tokenizer turns text into IDs before the model runs, and any information inside a token (such as spelling) is invisible unless the model memorized it.
• BPE builds vocabularies by merging the most frequent adjacent pairs, and byte-level BPE guarantees any input can be encoded, with no "unknown" tokens.
• Tokenization explains classic failure modes: letter counting, multi-digit arithmetic (75.6% vs. 97.8% on GPT-3.5 from commas alone), higher costs for some languages (up to 15 times longer encodings), and glitch tokens.
• The fix is usually to change the representation or hand off the work: split words into letters, deliberately format numbers, and call code for tasks that must be exact.
• Measure with the actual tokenizer. "4 characters per token" is a rough English average, not a budget.

Try it yourself: pip install tiktoken, then tiktoken.get_encoding("o200k_base").encode(" strawberry"), and see what comes out.