How Does Text Encoding Affect Language Models? UTF-8 Is Not a Universal Solution

Research reveals that UTF-8 is not optimal for all languages; encoding should be a language-specific design decision.
The article introduces a study that systematically examines the impact of text encoding on byte-level language model performance. UTF-8's variable-length design causes high-information-density languages such as Chinese and Korean to produce longer byte sequences for the same number of characters, introducing cross-lingual unfairness. The researchers propose the Aether hybrid encoding framework, which compromises between language-specific encoding and full UTF-8 coverage, and test it on four languages with markedly different scripts. The results show that high-information-density languages benefit significantly from compressed encoding, while low-information-density languages are insensitive to encoding choice. The authors therefore argue that text encoding should be a language-specific design decision rather than a universal default, and they offer practical implications for multilingual model developers.
The first step a language model takes when processing text is often converting characters into numbers. This seemingly neutral step actually harbors cross-lingual inequality. A recent study systematically examined the impact of text encoding on the performance of byte-level language models, and its conclusions are quite disruptive: UTF-8, as the de facto standard, is not optimal for all languages.

Why UTF-8's Variable-Length Design Creates Cross-Lingual Differences
UTF-8 uses variable-length encoding, and different scripts vary greatly in their byte-level representation length. Latin letters typically occupy only one byte, while characters such as Chinese and Korean often require three bytes. For language models that directly model byte sequences, this means that languages with high information density (such as Chinese and Korean) produce longer byte sequences for the same number of characters, and the model needs to handle more dispersed context.
This difference is not merely an engineering detail. The study points out that UTF-8's variable-length design introduces cross-lingual unfairness in the language model field, and the actual impact of encoding choice on model performance had previously hardly been systematically examined.
Byte-level language model refers to a language model that directly uses raw byte sequences (rather than tokenized tokens) as input and output units. Such models do not rely on a pretrained tokenizer and can theoretically cover any Unicode character or even binary data, making them naturally suitable for multilingual and open-vocabulary scenarios. But the cost is that the sequence length the model sees depends on the byte representation of the text; the same sentence can vary greatly in length under different encodings, which is the root reason why encoding choice affects modeling difficulty and computational cost.
Aether: A Hybrid Encoding Framework Proposed
To quantify the impact of encoding, the researchers proposed Aether, a hybrid encoding scheme: it uses script-specific encodings for the target language while maintaining full Unicode coverage through a UTF-8 fallback mechanism. With this framework, they compared UTF-8, other Unicode schemes, language-specific encodings, and custom encodings on four languages with markedly different scripts.
This design approach itself is noteworthy—it does not abandon Unicode's universality but seeks a compromise between universality and language characteristics.
Unicode fallback mechanism means that when a character is not in the character set of the selected language-specific encoding, the system automatically switches to another universal encoding such as UTF-8 to represent that character. This allows the core script of the target language to enjoy the short-sequence advantage brought by compressed encoding without losing foreign text, symbols, or rare characters due to incomplete encoding coverage. Aether relies on this fallback strategy to strike a balance between the efficiency of language-specific encoding and the universality of full Unicode coverage, avoiding the common problem of pure language-specific encoding schemes—"failure upon encountering an uncovered character."
Key Finding: Information Density Determines Encoding Sensitivity
The experimental results show a clear split. For languages with high character information density (Chinese, Korean), compressed encodings brought significant gains; for languages where individual characters carry less information (Georgian, Tamil), the impact of encoding choice was small, and the model was basically insensitive to the encoding method.
This finding challenges the default assumption that "one encoding fits all languages." It shows that the benefits of encoding optimization are strongly correlated with the information density of the script itself and cannot be generalized.
Character information density here refers to the average amount of semantic information carried by a single character. Scripts such as Chinese characters and Korean syllables often have one character corresponding to one morpheme or word, giving high information density; while in alphabetic scripts such as Georgian and Tamil, characters mostly correspond to phonemes, so the per-character information is lower. In byte-level modeling, high-density scripts, when represented in UTF-8, show more obvious byte expansion, causing the effective context to be stretched and information diluted; compressed encoding can alleviate this problem, so the gains for high-density languages are more significant. For low-density languages, the correspondence between characters and bytes is flatter, compression space is limited, and the marginal impact of encoding method is naturally smaller.
Encoding Should Be a Language-Specific Design Decision
Based on the above results, the researchers argue that text encoding should be a language-specific design decision, not a universal default. For languages with high information density, it is worth investing effort in designing compressed encodings; for languages with low information density, default UTF-8 may already be sufficient.
This has direct implications for the development of multilingual language models: when pursuing cross-lingual fairness, the encoding layer is an often overlooked but genuinely influential link. Rather than compensating for encoding-induced differences at the modeling stage, it is better to make more targeted choices at the encoding stage.
Implications for Practitioners
This study does not provide a simple answer to "which encoding is best," but rather an analytical framework and basis for judgment. For teams building multilingual models, this means that when supporting a new language, it is necessary to assess the character information density of that language and decide accordingly whether to adopt compressed encoding. Aether's hybrid approach also provides a reusable engineering template: enable custom encoding for specific languages while maintaining full Unicode coverage.
Encoding choice is not the most conspicuous link in language models, but this study shows that it deserves serious attention.
One-line Summary
Research reveals that UTF-8 is not optimal for all languages; encoding should be a language-specific design decision.
Paragraph Summary
The article introduces a study that systematically examines the impact of text encoding on the performance of byte-level language models. UTF-8's variable-length design causes high-information-density languages such as Chinese and Korean to produce longer byte sequences for the same number of characters, introducing cross-lingual unfairness to models. The researchers propose the Aether hybrid encoding framework, which compromises between language-specific encoding and full UTF-8 coverage, and compare them on four languages with markedly different scripts. The results show that high-information-density languages benefit significantly from compressed encoding, while low-information-density languages are insensitive to encoding choice. Accordingly, the authors argue that text encoding should be treated as a language-specific design decision rather than a universal default, and they offer practical implications for multilingual model developers.
Related articles

Why Microsoft CEO Nadella Still Backs Nvidia After Jensen Huang's Departure
Microsoft CEO Nadella publicly expresses confidence in Nvidia interim CEO Colette Kress. This article analyzes Nvidia's power transition after Jensen Huang's departure, Microsoft's compute dependency, and the trust logic of the AI supply chain.

Nadella: Critical Infrastructure Is 'Unpatchable' as AI Cybersecurity Arms Race Escalates
Microsoft CEO Nadella addresses AI cyberattack fears, saying much critical infrastructure is unpatchable, downplaying the Outlook outage, and analyzing AI's dual role in cybersecurity.

Musk Backs Jensen Huang: Firing Him Is Like Cutting Jordan
Musk says Jensen Huang is loyal to the U.S. and likens firing him to cutting Michael Jordan—highlighting politicization anxieties in the AI chip industry and defending talent value.