How Do LLMs Choose Writing Systems? Research Reveals a Last-Layer Commitment Mechanism

A Overlooked Multilingual Challenge: Script Selection

When we ask a large language model (LLM) to respond in Arabic, Hindi, or Chinese, the model must not only choose the right language — it must also select the correct writing system (script). A single language may map to multiple scripts, and choosing the wrong one renders the output completely unreadable. This paper published on arXiv (arXiv:2609.28784) focuses on a previously underexplored question: where is knowledge of writing systems distributed across model layers, and at what point does a model truly "commit" to using a particular script?

The researchers employed two complementary interpretability methods — logistic regression probing and logit-lens analysis — to analyze the model's internal representations layer by layer. These techniques can peer into the intermediate representations of each Transformer layer to determine where specific information is encoded and when it is finally acted upon.

Logistic regression probing is a classic technique in interpretability research: extract hidden state vectors from an intermediate Transformer layer, train a lightweight linear classifier, and test whether those vectors linearly encode some target category (e.g., "is the intended output script Arabic or Latin?"). If probe accuracy significantly exceeds a random baseline, the layer is considered to have encoded that information. Logit-lens takes a different approach: it projects hidden states from each layer directly into the vocabulary's logit space, asking "if we forced decoding at this layer, what would the model output?" — thereby tracking how predictions evolve across layers. The critical distinction is this: probing answers "does the information exist in this layer's representation?", while logit-lens answers "is this layer's representation sufficient to drive the correct final output decision?" The former reveals encoding; the latter reveals commitment.

Core Finding: Early Encoding, Late Commitment

The paper reveals a clear asymmetric structure. Probing experiments show that both the script of the input text and the output script specified in the instruction are encoded in the earliest layers of the network. In other words, the model "knows" which writing system to use very early on.

However, the actual commitment to an output script doesn't happen until the final few layers. Throughout most of the intermediate layers, the model's representations default toward Latin script. This suggests an internal inertia — a "Latin default" — that persists until just before the output stage, when the model finally switches to the target script.

This two-stage process is corroborated by logit-lens analysis: script commitment consistently occurs in the very last layers of the LLM. Two independent methods pointing to the same conclusion constitutes strong cross-validation.

The Latin default phenomenon arises because English and other Latin-script languages dominate LLM pretraining corpora, causing the model's intermediate-layer representation space to naturally skew toward Latin characters. This bias isn't a deliberate "choice" of Latin script — it's more of a statistical inertia. Before the final decision layers are reached, the highest-probability token candidates tend to come from the Latin character set. Similar phenomena have been noted elsewhere: multilingual models often default to English for internal "thinking" in chain-of-thought reasoning, even when the final output is in another language. This paper's contribution is to make that intuition precise — pinpointing exactly which Transformer layers exhibit the Latin bias and which layers complete the script switch.

Why Model Depth Matters

The study also found that smaller models perform worse on script-following tasks. Combined with the layer-by-layer analysis, the authors arrive at a convergent conclusion: script commitment is closely tied to model depth.

The logic follows naturally — if script decisions depend on representational transformations in the final layers, then shallower models with fewer layers lack the "space" to complete that transformation. This makes them more prone to errors in script selection, particularly for non-Latin writing systems. It also explains why small models in multilingual settings often "fall back" to Latin transliteration or simply output the wrong script.

Implications for Multilingual Architecture Design

The most practically relevant takeaway from this research is clear: building a truly inclusive multilingual model requires the network to be sufficiently deep. If script commitment naturally occurs in the final layers, and intermediate layers are persistently biased toward Latin, then adequate model depth is a prerequisite for supporting low-resource, non-Latin languages effectively.

For those pursuing lightweight or edge-deployed model compression, this is a signal worth heeding: aggressively reducing depth may disproportionately degrade output quality for non-Latin languages.

This finding carries specific warnings for model pruning and knowledge distillation in practice. Common compression strategies often uniformly reduce layer counts or prioritize preserving layers closest to the input or output. But if script commitment is heavily concentrated in a small number of final layers, uniform pruning will disproportionately harm non-Latin language output — while barely affecting Latin-script languages like English, which are implicitly supported throughout the middle layers. This suggests that compression schemes targeting multilingual use cases should preserve more capacity near the final layers, or use language-specific evaluation metrics (rather than English-only benchmarks) to properly measure the cost of compression.

Methodological Value

From an interpretability research perspective, this paper demonstrates how two orthogonal methods can be used to locate where a particular "capability" lives physically within a model. The probing method answers "does the information exist?", and logit-lens answers "when is the information used?" Together, they distinguish encoding from commitment — two concepts that are easily conflated.

This distinction is itself illuminating: a model "knowing" the answer in early layers does not mean it has "decided" to output that answer. A similar encoding–commitment separation may well exist for other multilingual and multimodal capabilities, and deserves further investigation.

Summary

Through rigorous interpretability experiments, this paper sketches the internal mechanism by which LLMs handle writing systems: script information is encoded early, a Latin default persists through the middle layers, final commitment is deferred to the last layers, and all of this is constrained by model depth. For researchers and engineers who care about multilingual fairness and inclusivity, it points to a clear design principle — without sufficient depth, genuine multilingual capability is hard to achieve.