Bridge-Railing Shadows and LLMs: Explaining Language Models with Geometric Principles

During an early morning run, the shadows cast by the carved stone balustrade of a roadside bridge made me stop in my tracks. The openwork carvings kept morphing in the sunlight—one moment they looked like swimming fish, a few steps later they became waves, and a little farther on, nothing remained but abstract, elongated patches of light.

This everyday scene maps neatly onto the core question of language models: the holes in the stone are concepts, the sunlight is context, and the shadows on the ground are the words we speak. In the paper The Simple Mathematics of Large Language Models, Joseph L. Breeden breaks down how Transformers work in plain language.

One Word, Three Shadows

Traditional NLP methods treat words as static symbols, assigning each a fixed vector. Breeden uses "bank" as an example:

• "I deposited money at the bank." (financial institution)
• "We walked along the river bank." (riverbank)
• "The plane banked sharply left." (tilting in flight)

If "bank" had only one fixed vector, it couldn't sit close to "money," "river," and "tilt" all at once. It's like looking at the bridge railing only at noon—you can only ever get one kind of shadow, yet linguistic meaning depends precisely on the angle of the light.

The Mathematics of Light: Attention as Weighted Averaging

Breeden's explanation strips away the anthropomorphic shell of Query/Key/Value and reduces the attention mechanism to three geometric steps:

1. The hole in the bridge itself (token embedding e) Before entering context, a word is just an initial embedding vector. It is the physical structure of the stone, containing every possible meaning, but not yet shaped by its environment.

2. The angle of sunlight (influence score aᵢ·bⱼ) The model projects the target word and context words through two learned matrices (W^A and W^B), computing a dot product to measure how well the "angle of light" matches the "shape of the hole." When the context is "deposited money," the light comes from the financial direction; when the context is "river," the light comes from the geographic direction.

3. The shadow on the ground (contextualized representation hᵢ) The model takes a weighted average of the context vectors (via a content projection W^C) to produce a brand-new vector hᵢ. The "bank" in the first sentence is pulled toward the financial region of the geometric space, while the one in the second is pulled toward the natural region. Same stone, completely different projection results.

The key to this explanation is that it removes the image of a "little person looking things up in a database." There is no lookup, only geometry and light. When the model reads "The plane banked sharply left," it doesn't "know" what a plane is; the high-dimensional vectors of "plane" and "sharply left" simply form a particular angle of light that passes through the "bank" hole and casts a shadow aligned with the geometric region of "physical motion."

Query/Key/Value (Q/K/V) are standard terms in the Transformer attention mechanism, originally drawn from an information-retrieval analogy: Query is "what I'm looking for," Key is "what label I can offer," and Value is "what I actually contain." Mathematically, all three are vectors obtained by applying different linear transformations (learned matrices) to the input embeddings; attention weights are computed from the dot product of Query and Key (normalized via softmax), and the output is a weighted sum of Values. Breeden's "geometry and light" framework is precisely an attempt to reduce this terminology back to pure vector operations: the so-called "query" is nothing but a dot-product match after projection, involving no explicit symbolic retrieval or rule-based lookup. Understanding this helps strip away the anthropomorphic metaphors and see attention for what it is: a differentiable, data-driven weighted-averaging operation.

Contextualized representations versus static word embeddings—the distinction between the two is key to understanding this article. Static embeddings (such as Word2Vec and GloVe) assign a unique vector to each word and cannot distinguish polysemous senses; contextualized representations, by contrast, are generated dynamically by the model based on the current sentence, so the same word receives different vectors in different contexts. ELMo was the first to systematically demonstrate this idea, and BERT and the GPT series brought it into the mainstream through Transformer attention layers. Breeden's "shadow" is exactly this kind of representation: it is not the word itself, but the result of the word being projected under a particular light (context). This also explains why LLMs don't need to store dictionary definitions—word meaning is computed on the fly at inference time.

Cross-Linguistic Differences in Bridge Holes

This metaphor also explains an interesting phenomenon: different languages "carve" concepts in different ways.

In English, "bank" is a multi-purpose bridge hole that requires complex dynamic lighting (the attention mechanism) to cast the correct shadow. Chinese, by contrast, has already carved three separate holes—银行 (bank), 岸 (shore), and 倾斜 (tilt). A Chinese model needs fewer "lighting operations" because the physical structure itself is already disentangled.

This is not to say which language is more "advanced," but rather to remind us that the structure of a language determines how much ambiguity resolution work the model has to do.

From a linguistic perspective, the phenomenon discussed in this article is closely related to lexicalization and semantic fields. Different languages partition the continuous conceptual space in different ways: English uses the single word "bank" to cover multiple senses—financial institution, riverbank, tilt—whereas Chinese lexicalizes them separately as 银行, 岸, and 倾斜. This difference in partitioning affects how hard the model has to work: languages with a high degree of lexicalization push ambiguity resolution down to the lexical level, while languages with less lexicalization rely more heavily on context. Note that this does not mean one language is "more efficient"; a model's disambiguation ability ultimately depends on the co-occurrence patterns in its training data, not on the inherent superiority of any language.

Meaning Is a Projection

The value of Breeden's paper lies not in proposing a new algorithm but in offering an intuitive framework for understanding LLMs. The model does not store dictionary definitions; it stores high-dimensional conceptual geometry. Meaning is not a static object you can hold in your hand, but a dynamic mathematical shadow cast when the light of context passes through linguistic structure.

Looking back at the bridge after my run, it suddenly struck me that it wasn't just casting shadows—it was performing a physical projection operation. And the beauty of language lies precisely not in the stone itself, but in the way it changes with the light.

One-line Summary

Breaking down the Transformer with a bridge-shadow metaphor: attention is light, and word meaning is a contextual projection.

Paragraph Summary

The article uses the metaphor of openwork bridge-railing carvings morphing in the morning sunlight to interpret Joseph L. Breeden's plain mathematical explanation of the Transformer mechanism. The holes in the stone represent concepts (token embeddings), the angle of sunlight represents context (influence scores), and the shadow on the ground is the model's output contextualized representation. Using the polysemous word "bank" as an example, the article shows that fixed word vectors cannot handle ambiguity, whereas the attention mechanism computes influence scores via dot products and then performs weighted averaging to dynamically generate context-dependent representations. The article further compares how English and Chinese carve up concepts, noting that language structure affects the amount of disambiguation work the model must do, and concludes that LLMs do not store dictionary definitions but instead build high-dimensional conceptual geometry, with meaning being the result of dynamic projection.