Animated Breakdown of GPT-4o's Tokenizer: Why It Can't Count the R's in Strawberry

The classic joke that large language models can't count the r's in "strawberry" actually hides some hardcore knowledge about tokenizers. Recently, a Reddit user created a 6-minute animation that fully traces the journey of "strawberry" through GPT-4o's tokenizer—from real token IDs to the BPE merging process—making it crystal clear why the model can't "see" letters.

One Word, Eight Tokens: The Real Segmentation Under o200k_base

The video opens with a counterintuitive fact: the question "How many r's are in strawberry?" is split into 8 tokens under o200k_base (the tokenizer used by GPT-4o and GPT-5). More importantly, the complete word " strawberry" with a leading space is itself a single token, with ID 101830. This means the three r's are packed into one ID, and what the model sees at the embedding layer is a single vector—not the ten characters s-t-r-a-w-b-e-r-r-y.

This also explains a common confusion: why the model sometimes gets the letter count right and sometimes wrong. When a word is merged into a single token, character-level counting becomes a task the model must accomplish "by detour"—it has to rely on similar patterns seen during training rather than actually counting.

o200k_base is the name of the tokenizer OpenAI designed for the GPT-4o series. Its vocabulary size is approximately 200,000 tokens—nearly double the cl100k_base (~100,000 tokens) used by GPT-3.5/GPT-4. The direct benefit of a larger vocabulary is that more common words (including complete words with leading spaces) can be encoded as single tokens, shortening sequence length and reducing inference cost. o200k_base significantly improves compression efficiency for non-English languages (such as Hindi and Chinese), which is the fundamental reason the Hindi example in the article drops from 21 tokens to 5 tokens. The tokenizer is open-sourced through the tiktoken library, and developers can directly call its encode/decode interfaces to reproduce the segmentation results in this article.

Why Subwords, Not Letters or Whole Words

The animation uses a custom text to demonstrate a miniature BPE (Byte Pair Encoding) run: starting from individual bytes, repeatedly merging the most frequent adjacent pairs until "strawberry" is split into subword combinations like st | raw | berry. The author explains the necessity of this compromise in the video—if split by letters, sequence length would become about 4 times longer; if split by whole words, the vocabulary would explode, and there would be no way to handle new words. Subword tokenization is exactly the balance point found between sequence length and vocabulary size.

As a detail, this demonstration is not a black-box result: every segmentation in the video comes from the actual output of running tiktoken 0.14.0.

Spaces Belong to Words, Not as Separators

An easily overlooked detail is that spaces do not exist independently in the tokenizer but stick to the preceding word. "strawberry", " strawberry" (with leading space), and "Strawberry" (capitalized) are three completely different token sequences in the tokenizer's eyes. This design allows the model to distinguish sentence-initial words from mid-sentence words, but it also means the same word may receive different token representations in different contexts.

BPE (Byte Pair Encoding) was originally a data compression algorithm, introduced to neural machine translation around 2016, and has since become the most mainstream tokenization method for large language models. Its training process is: first split the corpus into individual bytes (or characters), count the frequency of all adjacent symbol pairs, repeatedly merge the most frequent pair into a new symbol until the vocabulary reaches a preset size. The order and results of merges are solidified into a "merge rule table", which is applied in the same priority order during inference. Therefore, BPE segmentation is deterministic and reproducible—not something the model decides "on the fly". Understanding this is crucial: the token boundaries the model sees are products of statistical patterns in the training corpus, reflecting "which strings frequently appear together" rather than any linguistic notion of lexical structure.

Why Reasoning Models Are Better at Counting

The video cites an observation from OpenAI's 2025 paper "Why Language Models Hallucinate": DeepSeek^Ad-R1 spells out DEEPSEEK letter by letter before counting, while DeepSeek-V3 directly gives 2 or 3 (the correct answer is 1). This difference points to a larger issue—reasoning models use explicit chain-of-thought to break down the "counting" task into a form that can be operated on character by character, bypassing the limitation of single-token holistic representation.

This is not just a matter of being "smarter"—the reasoning process changes how the model accesses character-level information. When the model is guided to spell, decompose, and then count, it is actually constructing a path that does not depend on token merge results.

Chain-of-Thought (CoT) refers to the model generating a series of intermediate reasoning steps before giving the final answer. For tasks like counting, the significance of CoT goes beyond "thinking one step more": when the model writes out D-E-E-P-S-E-E-K letter by letter in its output, these letters are re-encoded as new input tokens, effectively "spreading out" the character information originally packed inside a single token into the sequence, allowing subsequent attention mechanisms to access them one by one. This is also why reasoning models perform better on character-level tasks—they actively reconstruct the granularity of the input through generation behavior, rather than relying solely on stronger parametric memory.

Tokenization Efficiency Differences in Multilingual Scenarios

The video ends with an additional example: the same Hindi sentence requires 21 tokens under GPT-4's tokenizer, but only 5 under GPT-4o (o200k_base). This order-of-magnitude difference shows that tokenizer version iterations have a significant impact on compression efficiency for non-English languages. Fewer tokens means lower inference cost and longer effective context—a tangible benefit for developers of multilingual applications.

From this perspective, "how many r's are in strawberry" is not just a meme—it's a small window into understanding the boundaries of large model capabilities: the world the model sees is made of tokens, not the characters we see.

The Value of Video as a Teaching Tool

The author states in the post that the narration is an AI voice reading a script they wrote, that the video is aimed at viewers learning about tokenizers, and that they actively seek correction feedback. This combination of "animation + real data + reproducible tools" makes it easier to build intuition than text explanations alone. For developers who want to dive deeper into BPE and tokenizer internals, this kind of content provides a low-barrier entry point.