Is a 2,103-Token Vocabulary a Mistake? A Vocabulary Size Experiment on Small Code Models

Introduction: When Vocabulary Size Becomes the Only Variable

When training small code models, the choice of tokenizer vocabulary size is often treated as a secondary hyperparameter. But an experiment report from the developer community raised a sharp question: Is a 2,103-token vocabulary wrong from the start?

The author had previously trained ORCH-Fusion (2,103 tokens, 272.7M parameters) and ORCH-Next.js-350M-v2 (16,000 tokens, 287M parameters), but the two differed in backbone, data, and training pipeline, making it impossible to determine whether the small vocabulary was actually holding things back. To address this, the new experiment treats the tokenizer as the sole variable: 12 LLaMA-style decoders, 3 model scales × 4 vocabulary sizes, with training FLOPs strictly aligned.

Experimental Design: A Fair Comparison Under Controlled Variables

The study used 7,606 TypeScript/JavaScript files that were precisely hash-deduplicated, and trained 4 byte-level BPE tokenizers with vocabulary sizes of 2k, 8k, 16k, and 32k respectively, all trained on the same byte sequences. For models, 3 LLaMA-style sizes were used with nominal labels of 10M, 25M, and 50M, though actual parameter counts differ due to embedding table differences. Each configuration was trained for only 1,000 steps, placing it in an undertrained state, so the conclusions apply only to vocabulary selection at this budget, not to the compute-optimal training regime. The evaluation metric is bits-per-byte (bpb) on a held-out set, which is insensitive to vocabulary size and suitable for cross-tokenizer comparison.

Byte-level BPE is the tokenization scheme used by models such as GPT-2. It first converts text into UTF-8 byte sequences, then runs the BPE merge algorithm at the byte level. Unlike character-level or word-level tokenization, byte-level schemes naturally cover any Unicode character without needing an UNK token, and vocabulary size can be set freely. The 2k, 8k, 16k, and 32k tokenizers in this experiment were all trained on the same byte sequences, with the only difference being the number of merge operations, ensuring a fair comparison.

Bits-per-byte (bpb) is a measure of a language model's compression efficiency: the model's negative log-likelihood on text divided by the original byte count. Since all tokenizers reconstruct to the same byte sequence length, bpb does not vary with token segmentation, making it suitable for cross-vocabulary comparison. Lower bpb means the model predicts text more accurately. Compared to perplexity, bpb avoids the artificial influence of vocabulary size on the metric.

How Vocabulary Size Affects Tokenization Efficiency

A tokenizer's "fertility" refers to the number of tokens per byte. The experiment measured:

Vocabulary Size	Tokens per Byte
2,000	0.4475
8,000	0.3667
16,000	0.3435
32,000	0.3275

Larger vocabularies significantly reduce fertility, meaning shorter sequences and faster inference. But the cost is embedding matrix expansion, consuming more FLOPs. The core question is: can the quality gain offset the computational cost?

Tokenizer fertility measures how many tokens, on average, each byte produces; a lower value means the same text is split into fewer tokens. Fertility directly determines sequence length: a fertility of 0.4475 means 1 byte of text requires about 0.4475 tokens, stretching the sequence; while 0.3275 compresses the sequence by about 27%. Shorter sequences mean fewer attention computations and autoregressive generation steps, yielding higher inference throughput. On the other hand, a larger vocabulary makes the embedding matrix (vocab_size × d_model) and output projection layer parameters grow linearly, consuming more FLOPs during training and potentially underfitting due to excess parameters.

Results: 8k–16k Vocabularies Win Across the Board

The bpb results on the held-out set are shown below (each cell is the evaluation value after 1,000 steps):

Non-embedding Parameters	2k	8k	16k	32k
10M	1.391	1.168	1.219	1.321
25M	1.378	1.258	1.218	1.280
50M	1.436	1.204	1.147	1.458

The 10M row is the mean of 2 random seeds, with seed noise as high as 0.13, so at the 10M scale, the difference between 8k and 16k can be considered a tie.

Size labels are nominal only; the actual parameter counts are as follows:

Size Label	Non-embedding Parameters	2k Total Parameters	8k Total Parameters	16k Total Parameters	32k Total Parameters
10M	3.9M	5.0M	8.0M	12.1M	20.3M
25M	13.3M	14.8M	19.4M	25.6M	37.9M
50M	31.5M	33.5M	39.7M	47.9M	64.2M

Key Findings

1. 8k–16k wins at all scales. The 2k vocabulary always performs worst; 32k collapses at the 50M scale because its embedding matrix occupies nearly half the model parameters and cannot be adequately trained within 1,000 steps.
2. Vocabulary choice matters more than backbone size. A 10M label with an 8k vocabulary (8.0M total parameters, bpb 1.168) beats a 50M label with a 2k vocabulary (33.5M total parameters, bpb 1.436)—4.2× fewer parameters yet 18.7% lower bpb.
3. Seed noise cannot be ignored. The bpb difference between two 10M seeds can reach 0.13, so differences smaller than about 0.1 should be honestly reported as ties.
4. A 2,103-token vocabulary is directionally wrong. At the 10M scale, the 2k vocabulary has about 19% higher bpb than 8k. However, ORCH also differed in data and training, so this is a suggestive conclusion, not definitive evidence.

Limitations and Next Steps

The experiment scale is limited: 3 sizes are insufficient to fit a scaling law; differences may fall within seed noise; at 10–50M parameters, the 32k vocabulary's embedding parameters dominate. Additionally, the RTX 3060's memory bandwidth limits throughput, requiring actual MFU measurement rather than guesswork.

Next steps include adding frontier API baselines, validating curves on a larger held-out set, and applying the findings to the next generation of ORCH models. The codebase has been open-sourced, with all 7 tests passing—the tokenizer is the only variable, and the conclusions are worth considering for small code model developers.

Model FLOPs Utilization (MFU) is a metric for training efficiency, defined as the ratio of actually achieved floating-point operation speed to the hardware's theoretical peak. On consumer-grade GPUs (such as the RTX 3060), memory bandwidth often becomes the bottleneck before compute, causing MFU to fall far below 100%. Because this experiment was limited by VRAM bandwidth, it could not accurately estimate the actual training FLOPs consumed, so the author plans to measure MFU empirically to more precisely compare computational costs across vocabulary configurations.

One-line Summary

In a controlled-variable experiment on small code models, 8k–16k vocabularies comprehensively outperform 2k and 32k, and vocabulary choice matters more than backbone size.

Paragraph Summary

A controlled-variable tokenizer experiment on small code models compared 3 model scales (nominal 10M/25M/50M) and 4 vocabulary sizes (2k/8k/16k/32k) fairly using bits-per-byte (bpb) as the metric. Results show 8k–16k vocabularies win at all scales; the 2k vocabulary is always worst, with about 19% higher bpb than 8k at the 10M scale; 32k collapses at the 50M scale because its embedding matrix occupies nearly half the parameters and is undertrained. Vocabulary choice outweighs backbone size: a 10M model with an 8k vocabulary has 4.2× fewer parameters than a 50M model with a 2k vocabulary, yet 18.7% lower bpb. The author notes the 2,103-token vocabulary is directionally wrong, but due to data and training differences, the conclusion is suggestive rather than definitive, and seed noise reaches 0.13.