The Geometry of Metaphor Representations in LLMs: Probeable and Steerable

Metaphor Is More Than Rhetoric: It's the Scaffold of Cognition

Metaphor is a core mechanism of human cognition—we use "time is money" to understand abstract concepts, and "temperature" to describe the closeness of interpersonal relationships. So what exactly happens inside a large language model (LLM) when it generates "a cold gaze" or "sharp criticism"? Does it truly "understand" metaphor, or is it merely imitating statistical patterns?

A study titled The Geometry of Metaphor: Probing and Steering Metaphor Representations in Large Language Models uses mechanistic interpretability tools to try to answer this question from a geometric perspective. The researchers not only locate the causal subspace related to metaphor in LLMs, but also demonstrate that the metaphoricity of model outputs can be controlled by intervening in these subspaces.

Research Method: Finding the "Causal Subspace" of Metaphor

The research team did not stop at observation; they employed intervention experiments. They trained linear probes on LLMs of various architectures and sizes (including models with different parameter counts and attention mechanisms) to distinguish metaphorical from non-metaphorical expressions. Crucially, these probes are not simple correlation detectors—the researchers further verified whether these directions have causal efficacy: activating or suppressing neurons along the probe direction systematically changes the model's classification judgments of metaphorical sentences.

The core of this methodology lies in treating metaphor as a linearly separable representational direction. If metaphor is a direction in the representation space, then "steering" along this direction should be able to modulate the model's behavior.

Effectiveness Across Parts of Speech and Tasks

The experiments covered metaphors of different parts of speech (nominal, verbal, adjectival), and interventions on the classification task all produced significant effects. This means that metaphor representation is not bound to specific lexical categories, but may be a more general semantic modulation mechanism.

Linear probe is a common tool in mechanistic interpretability research: a simple linear classifier (such as logistic regression) is trained on the activation vectors of a certain layer of the model to determine whether a concept (such as "metaphoricity") is linearly separable in that layer's representation. The probe itself only reveals correlation—a direction co-occurring with a concept does not mean the model actually relies on that direction for decision-making. Therefore, researchers further perform causal intervention: artificially increasing or decreasing activation values along the probe direction and observing whether the model output changes accordingly, thereby verifying the causal efficacy of that direction.

From Classification to Generation: Metaphoricity Can Be "Volume-Adjusted"

A more striking result is the success on the generation task. The researchers guided the generation process through activation vectors, causing measurable changes in the metaphor density of model outputs—equivalent to installing a knob on the model's "metaphor switch."

Potential applications of this capability include:

• Creative writing assistance: increase metaphoricity when poetic expression is needed, decrease it when technical documentation is required.
• Educational content generation: adjust the density of analogies in explanations according to the learner's age.
• Interpretability research: understand how LLMs organize abstract concepts by observing how metaphor representations interact with other semantic dimensions.

This also raises a deeper question: if metaphor is merely a steerable direction, is the model's "understanding" of metaphor essentially a kind of geometric computation rather than conceptual mapping in the human sense?

Activation steering is an inference-time intervention technique that does not require retraining the model: at each token generation, a scaled direction vector is added to the hidden state of a specific layer (e.g., h ← h + α·v), where α controls the intervention strength and v is the target direction. A positive α enhances the feature, while a negative one suppresses it. Unlike fine-tuning, activation steering takes effect immediately, allows sentence-by-sentence strength adjustment, and does not change model weights, making it particularly suitable for scenarios requiring dynamic control of writing style.

Language Models from a Geometric Perspective: Meaning as Direction

This work continues a main line of mechanistic interpretability in recent years: understanding semantic features as direction vectors in representation space. From "honesty direction" to "sentiment direction" and now to "metaphor direction," the internal representations of LLMs seem to exhibit a linearly decomposable structure.

But what makes metaphor special is that it inherently involves a mapping between two conceptual domains. How does a linear metaphor direction encode this cross-domain mapping? The paper's causal subspace method provides preliminary clues, but has not yet fully explained the generation mechanism of metaphor. In addition, models of different architectures and sizes all exhibit probeable metaphor subspaces, suggesting that this representation may be a universal feature that emerges naturally during training.

The linear representation hypothesis holds that high-level semantic features inside large models are roughly encoded as linear directions, with differences between concepts corresponding to vector differences and concept strength corresponding to the projection magnitude along that direction. This hypothesis provides theoretical support for "meaning as direction" and also explains why simple vector addition and subtraction can achieve concept manipulation. However, it mainly describes the separability of features, not the combinatorial logic between features—how cross-conceptual-domain mapping is achieved through the coordination of multiple directions remains a part not yet covered by this hypothesis.

Limitations and Future Directions

Current controllability experiments are mainly conducted in English contexts, and whether metaphor representations across languages share the same geometric structure remains an open question. In addition, the cultural specificity of metaphor—such as the difference between Chinese "吃醋" (eat vinegar, meaning jealousy) and English "green-eyed"—may imply that there are multiple metaphor subspaces inside the model rather than a single direction.

The most practical insight from this research may be: we do not need to treat the metaphor ability of LLMs as a black box. With geometric tools, we can locate, measure, and adjust it. For AI application developers who need fine-grained control of writing style, this provides an internal intervention path beyond prompt engineering.

Paper information: The Geometry of Metaphor: Probing and Steering Metaphor Representations in Large Language Models. Source: arXiv (rss).

One-line Summary

The study uses geometric tools to locate the metaphor subspace of LLMs and achieves controllable generation of metaphoricity.

Paragraph Summary

This article introduces a mechanistic interpretability study on metaphor representations in large language models. Researchers trained linear probes on LLMs of various architectures and sizes to distinguish metaphorical from non-metaphorical expressions, and verified through activation interventions that these representational directions have causal efficacy. Experiments show that metaphor representations can be probed and steered in classification tasks across different parts of speech, and that in generation tasks, activation steering can change the metaphor density of outputs, enabling "volume-adjustment"-style controllable generation. The article notes that this continues the research line of treating semantic features as direction vectors in representation space, but also leaves open questions such as cross-domain mapping mechanisms, cross-lingual sharing, and cultural specificity, and provides an internal intervention path beyond prompt engineering for AI writing style control.