[KongchangAI]
· 2 min read· 1,341 words

Do Position Codes Determine Left vs. Right? Counterfactual Experiments on Qwen and InternVL Vision Models

Do Position Codes Determine Left vs. Right? Counterfactual Experiments on Qwen and InternVL Vision Models

Counterfactual experiments show vision-language models' left/right judgments rely mainly on ViT position codes, not LLM token order.

This study uses counterfactual experiments mirroring ViT width-direction position codes to probe how vision-language models judge left/right. On Qwen2.5-VL, Qwen3-VL, and InternVL3, mirroring position codes reproduces 84.3%-93.2% of flipped-image answers, while LLM-side ID and token-order mirroring is nearly ineffective; the effect concentrates in the ViT's final global-attention block. On What'sUp, however, above/below and support relations rely more on content orientation and token order, showing different spatial relations use different pathways. Most remaining errors are relation-binding failures, suggesting repair strategies must be tailored by relation type.

How do vision-language models decide what's "left" and what's "right"? Counterfactual experiments on Qwen2.5-VL, Qwen3-VL, and InternVL3 yield a surprising answer: left/right orientation likely comes primarily from the position codes of the vision transformer (ViT), not from token order or ID logic on the language-model side.

rss source: Where Do Vision-Language Models Get Left and Right? Position-Code Counterfactuals in Qwen2.5-VL, Qwe

Three Candidate Pathways: Content Orientation, ViT Position Codes, and LLM Token Order

The researchers open by proposing three candidate mechanisms by which vision-language models judge left/right orientation: the orientation of image content within patches, the ViT's position codes, and the token positions and order in the language model (LLM). Ablation experiments have already shown that ViT position codes are a necessary condition—but "necessary" does not equal "sufficient." To that end, the authors designed a finer-grained test: mirror only the position codes and see whether that alone can flip the model's left/right answers.

If mirroring the position codes is enough to reproduce "the answer after flipping the image," that shows position codes play a dominant role in this decision pathway. This design advances beyond "necessary condition" to a test of "sufficiency," and it is a key step toward understanding the internal mechanisms of spatial reasoning in multimodal models.

Vision Transformer (ViT) and Position Codes

A ViT splits an image into fixed-size patches; each patch is linearly projected into a feature vector. However, self-attention itself is order-insensitive, so positional information must be injected separately. Common approaches include learned absolute position codes (as in the original ViT) or 2D-interpolated sinusoidal position codes (as used by Qwen2.5-VL), where the latter generates codes separately along the width and height directions and sums them. These position codes are fused with content features in every ViT block, enabling the model to distinguish "left patches" from "right patches." Mirroring the position codes along the width direction is equivalent to telling the model "the image has been flipped left-right," while the pixel content remains unchanged—thus allowing the causal effect of position codes on left/right judgments to be tested in isolation.

Mirroring ViT Width Positions Flips the Vast Majority of Left/Right Answers

The experiments were run on Qwen2.5-VL-7B, Qwen3-VL-4B, and InternVL3-2B. After mirroring the ViT's width-direction positions, the models reproduced the flipped-image answers on CV-Bench left/right questions at a rate of 84.3%–93.2%; Qwen2.5-VL-3B reached 87.5%, though some of this reflects a fallback toward the "left" direction.

Since the questions have only two options, this rate can also be read as the proportion of answers that were perturbed. The researchers further found that shifting the position grid by half a cell likewise flips most answers. This shows that the influence of ViT position codes on left/right judgments is not marginal but directly decisive.

Mirroring on the LLM Side Is Nearly Ineffective

In sharp contrast to the ViT side, when the ViT position codes were held fixed, mirroring IDs and mirroring token order on the LLM side changed only a tiny fraction of left/right answers across the three model families. The ID test in Qwen2.5-VL was itself relatively weak, yet even so no notable effect was observed. This asymmetric result further points the "responsibility" for left/right judgments toward position codes on the vision side.

The Effect Concentrates in the Global-Attention ViT Block, Mainly the Last Layer

On Qwen2.5-VL-7B, the researchers localized the source of the effect: it concentrates in the ViT's global-attention blocks, and mainly in the final block. This finding has direct implications for model repair—if left/right judgments need to be intervened on, the target layer should be located in this region, not in token processing on the language-model side.

rss source: Where Do Vision-Language Models Get Left and Right? Position-Code Counterfactuals in Qwen2.5-VL, Qwe

Global Attention vs. Local Attention

In modern ViT architectures such as Qwen2.5-VL, attention is often divided into global blocks and local blocks: global blocks allow all patches to interact pairwise and capture long-range dependencies, while local blocks are restricted to windows and focus on fine-grained features. Judging left/right relations requires comparing content on both sides of the image, so it naturally depends on global attention. The researchers found the effect concentrated in the final global block, suggesting that this layer may play a key role in integrating positional information into spatial-relation representations. This also explains why merely mirroring position codes can substantially flip answers—the position signal in the final layer directly participates in outputting the left/right decision.

A Different Picture on What'sUp: Content and Token Order Matter More

On the What'sUp dataset, the conclusions diverge. Flipping content and token order—without flipping position codes—preserves more "on/under" answers than flipping position codes does, and stays closer to the answers after flipping the image. This means different spatial relations rely on different mechanisms: left/right depends more on ViT position codes, whereas above/below and support relations are more sensitive to content orientation and token order.

This contrast reminds us that "spatial reasoning" cannot be treated as a single capability. Different relation types may travel through different information pathways, and model-repair strategies need to be differentiated by relation category.

The What'sUp Dataset and Spatial Relation Types

What'sUp is a visual question answering benchmark focused on spatial relation understanding. It includes relation types such as "left/right," "above/below," and "supporting/supported," and its questions typically involve the relative positions of two objects. Unlike the simple left/right binary choice in CV-Bench, What'sUp's relation judgments rely more on understanding object orientation and frames of reference. For example, "the cup is on the table" requires recognizing a support relation, whereas "the cup is to the left of the table" relies more on horizontal position comparison. This difference in task design makes different relation types vary in their sensitivity to position codes, content orientation, and token order—leading to the divergent conclusions observed in the experiments.

Caveats from the Post-Hoc Analysis

The authors conducted several post-hoc analyses of the above results and honestly presented their complexity. On CV-Bench questions with localized objects, random permutation flipped 18.9–31.0 percentage points more answers when reversing object order than when preserving it. On What'sUp, in Qwen2.5-VL and Qwen3-VL, the proportion of left/right changes caused by random permutation that land on the mirror relation was mostly below chance level. Measured as "returning to the original answer," the contrast between support relations and left/right relations on What'sUp was clearly visible only in InternVL3-2B.

Among the remaining errors, in most cases the model gives the same relation no matter which object is asked about—suggesting that the repair direction may need to target relation binding, object localization, or the use of the reference object, rather than simply adjusting position codes.

Implications for Model Repair and Evaluation

The value of this work lies not only in localizing a mechanism but also in providing an actionable diagnostic framework. Multi-source cross-validated ablations and counterfactual tests show that ViT position codes are the main carrier of left/right judgments, but this conclusion is constrained by dataset and relation type. For developers, when facing failures in spatial reasoning, first distinguishing whether the problem is left/right-type or above/below-type, and then deciding whether to adjust visual position codes or improve object binding, may be more effective than broadly "adding data." The evaluation side warrants caution as well: conclusions from a single dataset should not be directly extrapolated to all spatial relations.

One-line Summary

Counterfactual experiments show that vision-language models' left/right judgments rely primarily on ViT position codes, not on language-model token order.

Paragraph Summary

This study uses counterfactual experiments that mirror ViT width-direction position codes to probe the mechanistic source of left/right orientation judgments in vision-language models. On Qwen2.5-VL, Qwen3-VL, and InternVL3, mirroring position codes reproduces 84.3%–93.2% of flipped-image answers, while ID mirroring and token-order mirroring on the LLM side are nearly ineffective; the effect concentrates in the ViT's final global-attention block. On the What'sUp dataset, however, above/below and support relations rely more on content orientation and token order, indicating that different spatial relations follow different information pathways. Most remaining errors manifest as relation-binding failures, suggesting that repair strategies need to be differentiated by relation type.

#vision-language models#ViT position codes#Qwen2.5-VL#Qwen3-VL#InternVL3#counterfactual experiments#spatial reasoning#left-right judgment
Share:

Related articles