The Mystery of Spatial Understanding in VLMs: Why Representations Are Less Reliable Than Lexical Priors

When VLMs "Destroy" What They Already Know

Vision-Language Models (VLMs) are being increasingly deployed for image analysis and reasoning tasks. Yet a long-overlooked question remains: do the internal representations of these models faithfully encode the structured relationships needed for reasoning? A new study offers a sobering answer — at the representation level, VLMs encode spatial relationships significantly less faithfully than lexical information, even when their output scores on compositional benchmarks are well above chance.

In other words, a model that "gets the answer right" doesn't necessarily "understand" the spatial relationships between objects in an image at an internal level. This failure is completely invisible to traditional output-based evaluation, yet it can introduce hidden risks in downstream tasks.

Two New Tools: EICS and CASP-VL

To expose this hidden problem, the researchers introduce two methodological tools.

EICS: Evaluation-Invariant Concept Stability

EICS (Evaluation-Invariant Concept Stability) is a metric for probing representations. Its core idea is to test whether a model's representations consistently encode a target concept under controlled "single-concept interventions." If a representation truly captures a concept, changing only that one concept should produce predictable, identifiable changes in the representation. This invariance test is more revealing of the true state of internal representations than simply examining model outputs.

CASP-VL: Causally Controlled Spatial Perturbations for Vision-Language Models

CASP-VL (Causally controlled Spatial Perturbations for Vision-Language models) is a benchmark dataset specifically designed for spatial relations. Built on Visual Genome scene graphs, it introduces spatial perturbations in a causally controlled manner. The structured annotations provided by scene graphs allow researchers to precisely manipulate spatial predicates in image descriptions, enabling clean measurement of a model's spatial relation encoding without interference from other variables.

Visual Genome is a large-scale visual knowledge graph dataset released by Stanford University, containing over 100,000 images, each annotated with dense scene graphs. Scene graphs represent image content in a structured way: nodes represent objects, and edges represent relationships between objects (including spatial predicates such as "above/below/left of," as well as actions and attributes). This structured representation allows researchers to precisely substitute a relational predicate — for example, changing "cup on table" to "cup under table" — without altering the main content of the image, enabling single-variable controlled interventions. Compared to constructing contrastive samples directly from natural language descriptions, this scene-graph-based approach effectively prevents other semantic variables from contaminating the results, making measurements more causally interpretable. This is the key infrastructure that enables CASP-VL to "cleanly" measure spatial relation encoding capability.

Core Finding: Spatial Relations Lag Behind Lexical Priors

Experiments across multiple VLM architectures reveal a consistent pattern: the fidelity with which spatial relations are encoded in representations is significantly lower than for lexical information.

This contrast is particularly striking. These models perform reasonably well on compositional benchmarks, with scores well above chance. By traditional output-based evaluation logic, they appear to have "mastered" spatial reasoning. Yet once you look beneath the surface to the representation level, it becomes clear that models are not retaining relational information in the representations actually used for downstream computation.

The study uses "lexical priors" as a point of comparison — models encode lexical-level information relatively robustly, while spatial relations emerge as a clear weak point. This suggests that on many tasks where models appear to "understand space," they may be relying more on lexical statistical shortcuts than on genuine representations of spatial structure.

In this context, "lexical priors" refer to word co-occurrence statistics that models learn from large-scale pretraining corpora. For example, a model exposed to countless descriptions of "cup on table" will tend to encode the association between "cup," "on," and "table" as robust features — even if the actual relationship depicted in the image is different. This reliance on lexical statistical shortcuts rather than genuine visual spatial reasoning is difficult to detect through output-based accuracy evaluation, because such shortcuts often happen to yield the correct answer. Compositional benchmarks are specifically designed to break these shortcuts by constructing semantically unusual combinations to test models — yet as this study shows, even when models answer correctly on such tests, their internal representations may not actually encode the corresponding spatial structure.

Where Is the Information Lost? Aggregation Is the Culprit

Further analysis pinpoints where the problem occurs. The study finds that spatial relations are identifiable at the token positions of the predicates themselves — meaning relational information does exist within the model at some point.

But the problem lies at the aggregation stage: when the entire sentence is compressed into a single global representation, relational information is lost along the way. Whether the aggregation uses the end-of-sequence (EOS) token or mean pooling, the result is the same — the relational information that should have been preserved disappears from the global representation.

This reveals a structural flaw: relational information was present in the model's input and was encoded at local token positions, yet when generating the global representation used for downstream tasks, this information gets "diluted" or "erased." The model effectively "destroys what it already knows" — which is precisely where the paper's title comes from.

EOS (End-of-Sequence) token aggregation and mean pooling are currently the two most common methods for generating global sentence vectors from sequence representations. In Transformer-based language and vision-language models, the EOS token typically aggregates global context information through self-attention, serving as a "summary" vector for the entire sequence — widely used in contrastive learning (e.g., CLIP) and retrieval tasks. Mean pooling takes the arithmetic average of the hidden states of all tokens and is another simple, efficient sentence representation scheme. These two methods typically perform similarly in practice, but this study shows they share the same structural weakness: when it comes to scenarios requiring precise preservation of predicate relations, the aggregation operation itself causes information loss. This finding suggests the root cause is not any particular aggregation algorithm, but rather that existing aggregation paradigms lack active mechanisms to protect relational information.

Implications for Research and Applications

The most important contribution of this work is exposing a class of systematic failures that are completely invisible to output-based evaluation.

For a long time, the community has relied on benchmark accuracy to measure VLM reasoning ability. But this study shows that high accuracy can mask fundamental flaws at the representation level. A model can answer spatial reasoning questions correctly while failing to stably retain spatial relations in its internal representations — this is a latent risk for any downstream system that depends on those representations.

For researchers, EICS and CASP-VL offer a new evaluation paradigm that goes beyond the output layer to directly examine representation quality. For engineers building real-world applications on top of VLMs, this is a clear reminder: a model's "capability" and its "representational fidelity" are two different things — especially for tasks involving fine-grained spatial relations, one should not draw optimistic conclusions about representational ability from benchmark scores alone.

A promising direction for future work is redesigning representation aggregation mechanisms so that relational information is not destroyed as it is compressed from local tokens into a global representation.