Visualizing LLM Poetry Generation: Why Chinese Causal Structures Look More Like Matrices

From English Poetry to Chinese Poetry: An Intriguing Visual Exploration

What exactly happens inside a large language model (LLM) when it writes a poem? Which tokens influence the generation of other tokens? Questions like these about the internal flow of information in models have long been central to interpretability research.

Recently, a developer shared a fascinating experiment on Reddit. Having previously published work on visualizing the causal structure of English poetry generation, they applied the same methodology to Chinese poetry — and arrived at a surprising conclusion: the causal dependency graph (DAG) for Chinese poetry looks more like a neat matrix than the relatively loose, tree-like structure seen in English.

This visual difference is no mere aesthetic coincidence. It reflects deep structural differences in how the two languages organize characters and handle rhythmic structure.

Methodology: Tracing LLM Generation with Causal Graphs

What Is a Causal Structure Graph?

A causal structure graph extracts the dependency relationships that a model relies on when generating each token — specifically, which previous tokens influenced it — and presents them as a directed acyclic graph (DAG). Nodes in the graph represent individual characters or tokens, while edges represent attribution relationships: who influenced whom.

The model used in this experiment is Qwen3-1.7B — a relatively small model that handles both generation and attribution analysis. Even at this scale, the resulting causal graphs are remarkably intuitive and consistent with human linguistic intuition. This suggests that the structural regularities of language have stable representations inside models, and don't entirely depend on massive parameter counts.

Attribution relationships are typically computed using gradient-based methods or attention weights. Common approaches include computing the gradient norm of each output token with respect to input tokens (gradient-based attribution), or directly aggregating the weight matrices from multi-head attention. The DAG is "directed" because language generation has a unidirectional temporal dependency — later-generated tokens can depend on earlier ones, but not vice versa. It is "acyclic" to ensure that the graph's topological order aligns with the generation order, making it easy to trace how information propagates from the prompt through to the final output. Unlike attention heatmaps, the DAG format more clearly reveals multi-hop dependency chains — for example, how a key imagery token first influences a rhyme token, which then indirectly constrains the wording of an entire line.

Cross-Linguistic Consistency in Rhythm and Boundary Tokens

One cross-linguistic pattern the author observed was that rhythm and boundary-token behavior are essentially the same in both Chinese and English.

Boundary tokens here refer to special tokens at positions like punctuation marks, line breaks, and stanza separators. When generating poetry, models show a clear concentration of attention on these boundary positions, because they mark the beginning and end of rhythmic units. This behavior is highly consistent across Chinese and English, suggesting that the model applies a unified, cross-linguistic logic when processing the rhythmic skeleton of poetry.

Why Chinese Poetry's Causal Graph Looks More Like a Matrix

The Grid-Like Layout of Chinese Characters

The truly interesting difference lies in the overall visual form. The author notes that Chinese characters are arranged on the page more like a grid, so the corresponding DAG reads more like a matrix.

This warrants a deeper look:

• Higher information density in Chinese: Each Chinese character is a relatively complete semantic unit. A classical quatrain (绝句) typically follows a strict equal-length structure of five or seven characters per line. This uniformity in character count naturally produces regular row-and-column relationships.
• Variable token length in English: English words are composed of multiple subword tokens of varying lengths. Sentences are more irregular in both visual appearance and structure, so causal graphs tend toward tree-like or irregularly branching forms.

In other words, Chinese poetry already forms a two-dimensional matrix layout in terms of "number of lines × characters per line," and the model's causal dependencies flow naturally within this regular framework — which is why the resulting graph takes on such a matrix-like regularity.

Subword tokenization is the standard approach modern LLMs use to process English, with representative algorithms including BPE (Byte Pair Encoding) and WordPiece. For example, "poetry" might be split into "poet" + "ry" as two tokens. Chinese characters, by contrast, have a limited character set and each character carries independent meaning, so in the vast majority of cases a single Chinese character maps to a single token. This fundamental difference means that a twenty-character five-character quatrain (五言绝句) produces a token sequence of exactly 20 (plus punctuation) in Chinese, whereas an English translation processed with subword tokenization often expands to 30–50 tokens of uneven lengths. This "one character, one token" regularity is precisely what causes the causal dependency matrix for Chinese poetry to align naturally along row and column dimensions, producing a visually regular grid pattern.

Implications for AI Interpretability Research

The significance of this finding is that a language's formal structure maps directly onto the shape of information flow inside a model. For highly structured texts like classical quatrains, the regularity of Chinese makes causal relationships easier to observe and generalize. This is good news for researchers — metrical poetry can serve as a kind of natural probe for testing whether a model has genuinely internalized the structural constraints of verse.

"Probing" is one of the key methodologies in interpretability research: researchers select texts with known, quantifiable structure, and examine whether the model's internal representations encode that structure during training or inference. Metrical poetry — whether Chinese quatrains or English sonnets — naturally constitutes a class of "fully structure-annotated" probe texts due to its strict constraints on character count, rhyme, and tonal patterns. Researchers can use these metrical rules as a reference baseline without any additional annotation. Free-form prose, by contrast, has blurry structural boundaries, making the regularities in causal graphs much harder to assess. This explains why poetry appears far more frequently than other genres in LLM interpretability research.

InfoLens: An Open-Source Tool for Reproducible Interpretability Experiments

One of the most commendable aspects of this work is that the author has open-sourced the entire toolkit and provided an interactive online demo.

• Online Demo: Users can enter prompts directly (e.g., "write a quatrain about spring") and observe the causal flow graph as the model generates text in real time.
• Source Code: The project is called InfoLens, hosted on GitHub, and anyone can reproduce the full attribution pipeline locally.
• Accompanying Video: The author also provides a demonstration video showing the dynamic generation process of causal graphs.

This combination of a small model, open-source tools, and an interactive demo significantly lowers the barrier to interpretability research. Work of this kind used to depend on large models and complex analysis pipelines. InfoLens demonstrates that with a 1.7B model, you can produce intuitive and compelling visualizations.

Closing Thoughts: Visualization Makes the Model's Interior Visible

Though modest in scale, this experiment offers a genuinely illuminating perspective: by visualizing causal structures, we can directly perceive the "morphological differences" between languages inside a model.

The matrix-like quality of Chinese poetry and the tree-like quality of English poetry are, at their core, projections of linguistic structure onto neural network information flow. As interpretability tools become increasingly accessible, similar methods may one day help us examine models' internal mechanisms for more complex tasks like translation, reasoning, and code generation.

For developers and researchers interested in AI interpretability, lightweight open-source tools like InfoLens are well worth trying — they make "opening the model's black box" genuinely within reach.