Programming-Language-Aware Tokenization: The Key to Better Generalization in Code Models

Why Code Model Tokenizers Need to "Understand" Programming Languages

When large language models process code, the tokenizer is often an overlooked component. A developer shared his experiment on Reddit: while developing a software engineering model for his personal agent, he found that the tokenizers of Nvidia Nemotron 3 and Qwen 3.5 preserve most operators and delimiters, but sometimes split keywords or operators at awkward positions. For example, the <<= operator might be split into multiple different tokens such as <<, = or <, <=.

Programming languages have their own syntax, semantics, structure, and "grammar," and keywords and operators carry semantic meaning. A basic unit is called a lexical token, identified by the language's lexer. When a lexical token is split into multiple tokens or merged together, the meaning or behavior may change completely. The model has to additionally learn "these separate fragments actually form a single unit" or "this merged vocabulary entry actually has an independent meaning." This extra learning burden directly affects the model's ability to understand and generate code.

Lexical Token is a fundamental concept in the compilation pipeline of programming languages. Before source code enters syntax analysis, it is first processed by a lexer and split into a series of meaningful atomic units, such as keywords (if, function), identifiers, operators (<<=, +=), literals, and delimiters. Each lexical token is the smallest semantically meaningful unit defined in the language specification, and the compiler's subsequent syntax tree construction relies entirely on the correctness of these boundaries.

It is worth noting that the algorithms used by LLM tokenizers, such as BPE (Byte Pair Encoding) or SentencePiece, have different goals from the lexical rules of programming languages: the former aims to compress text based on statistical frequency, while the latter aims for syntactic correctness. This causes the tokenizer to possibly split <<= into << and =, or cut function into func and tion, breaking the integrity of lexical tokens. The model therefore needs to additionally learn "fragment recombination rules," adding unnecessary learning burden.

Experimental Design: Vocabulary Expansion + Fine-Tuning Comparison

To preserve lexical token boundaries, the developer took a direct approach: expanding the vocabulary of the Qwen3.5-4B-Base model to add missing keywords from languages such as Python, C++, Golang, and Rust. He then fine-tuned the vocabulary-expanded model on the SWE dataset.

For a fair comparison, he also fine-tuned the original base model (using its own tokenizer) with the same recipe. Both models were trained on the same training data and pipeline, with the only variable being whether the tokenizer underwent programming-language-aware expansion. This controlled-variable experimental design allows performance differences to be attributed to the tokenizer improvement.

Vocabulary Expansion refers to the technique of appending new tokens to the tokenizer of an existing pretrained model. The specific operations usually include: identifying strings that are high-frequency in the target domain but not covered by the existing tokenizer (such as programming keywords), adding them to the vocabulary as independent tokens, and simultaneously expanding the model's corresponding embedding layer and output layer parameters. New parameters are typically initialized using the mean of existing token embeddings or random initialization.

Fine-tuning is needed after vocabulary expansion because the embedding vectors of the new tokens have not been pretrained, and the model needs to learn their representations in downstream tasks. In this experiment, fine-tuning on the SWE dataset both gives the new keyword tokens meaningful embeddings and adapts the model to the new tokenization boundaries. The advantage of this approach is that it preserves most of the original model's capabilities while introducing domain adaptation at a relatively low cost.

Performance Gains: HumanEval Rises Significantly, MBPP Declines Slightly

The experimental results show clear performance changes. The vocabulary-expanded model's pass rate on HumanEval increased from 45.7% to 52.4%, and on HumanEval+ from 39.0% to 47.0%. These two benchmarks mainly evaluate the model's ability to generate correct code based on function signatures and docstrings, with improvements of 6.7 and 8.0 percentage points respectively, indicating that programming-language-aware tokenization indeed helps the model generalize better to unseen code tasks.

However, on the MBPP (Mostly Basic Programming Problems) benchmark, the Evalplus version score slightly decreased from 65.3% to 64.6%. MBPP problems are usually shorter and more basic, and the impact of tokenizer changes may differ. This slight decline is worth noting, but overall, the substantial gains on the HumanEval series are more convincing.

HumanEval and MBPP are two widely used code generation benchmarks, but they have different design orientations. HumanEval, released by OpenAI, contains 164 hand-written programming problems, each providing a function signature and docstring and requiring the model to complete the function body, with the evaluation metric being pass@k (the probability that at least one of k generated samples passes the unit tests). HumanEval+ is an enhanced version by the EvalPlus team that expands the number of test cases and can more strictly detect code correctness.

MBPP (Mostly Basic Programming Problems), on the other hand, contains about 1,000 entry-level programming problems with shorter descriptions and more basic tasks. The differences in difficulty and problem types between the two can explain why tokenizer improvements significantly boost HumanEval but have limited or even slightly negative effects on MBPP: HumanEval problems involve more complex combinations of operators and keywords, where the integrity of lexical boundaries is more critical; while MBPP problems are short, and the model is less sensitive to tokenization fragments.

The Underlying Logic: Lexical Boundaries and Model Learning Burden

Why does preserving lexical token boundaries bring such improvements? The core lies in reducing the "patchwork rules" the model needs to learn. When the tokenizer splits <<= into << and =, the model must learn that these two tokens combined represent the left-shift assignment operator. Such combination rules may not be common enough in the training data, causing the model to make mistakes during generation. But if <<= itself is a token, the model can directly learn its semantics without extra combination.

Similarly, splitting keywords also increases learning difficulty. For example, if function is split into func and tion, the model needs to understand that these two fragments together form a keyword. Expanding the vocabulary adds these missing keywords as independent tokens, allowing the model to directly process complete lexical units.

This approach is consistent with the design philosophy of programming languages: lexical analysis is the first step in the compilation process, and identifying meaningful tokens is the foundation for subsequent syntax analysis. Having the model's tokenizer simulate this process is equivalent to providing a representation at the input level that better conforms to the language structure.

Practical Implications and Resources

The developer has publicly released the experimental details, code, model weights, and checkpoints. The blog address is https://www.soumyaranjansahu.com/blog/qwen35-swe. For developers who want to reproduce or learn from it, this is a valuable resource.

From a broader perspective, this experiment suggests that in addition to pursuing larger models and more data, optimizing input representations is equally important. Programming-language-aware tokenization is a low-cost, high-return improvement direction, especially for code generation and software engineering tasks. In the future, more work may explore how to incorporate the lexical, syntactic, and even semantic structures of programming languages into tokenizer design to further improve the generalization ability of code models.

Of course, this experiment is based on a single model and specific datasets, and the generality of the conclusions requires more validation. The slight decline on MBPP also reminds us that tokenizer changes may involve trade-offs. But the significant improvement on HumanEval is enough to show that making tokenizers "understand" programming languages is a path worth exploring in depth.

One-line Summary

Expanding the vocabulary to preserve lexical token boundaries in programming languages improves the code model's HumanEval score by 6.7 percentage points.

Paragraph Summary

The article introduces an experiment shared by a developer on Reddit: addressing the problem that code model tokenizers often split keywords and operators at awkward positions, he proposed a programming-language-aware tokenization strategy to preserve lexical token boundaries. The experiment expanded the vocabulary of Qwen3.5-4B-Base to add missing keywords from languages such as Python, C++, Golang, and Rust, fine-tuned on the SWE dataset, and compared it with a model using the original tokenizer under controlled variables. Results showed that the vocabulary-expanded model improved from 45.7% to 52.4% on HumanEval and from 39.0% to 47.0% on HumanEval+, but MBPP slightly decreased by 0.7 percentage points. The author believes that preserving lexical boundaries can reduce the model's burden of learning "fragment patchwork rules" and is a low-cost, high-return improvement direction. The experimental code and model weights have been released.