Why Vision-Language Models Are Fooled by 2D Images: CueClash Benchmark Reveals 3D Spatial Reasoning Shortcomings

CueClash benchmark reveals frontier VLMs rely on 2D image shortcuts rather than true 3D layout, with shortcut gaps of 18-37 points.
The new CueClash benchmark renders scenes so that 2D plane cues for five spatial relations are correct in exactly half of the statements and wrong in the other half, separating models' reliance on 2D shortcuts from true 3D layout. Results show that Claude Opus 5.5, GPT-5.5, and Gemini 3.6 Flash suffer significant accuracy drops when 2D cues are wrong, with shortcut gaps of 18-37 points, and 7-29 points even after excluding confusing relations. Yet with direct ground coordinates, all three achieve 99%-100% accuracy, indicating complete conceptual knowledge and a bottleneck in image-to-layout conversion. Model-generated top-down maps preserve depth ordering but compress depth: GPT-5.5 nearly copies the image, while Opus 5.5 and Gemini partially correct. Real photos on CV-Bench confirm the same misleading 2D cue effect.
Title
Why Vision-Language Models Are Fooled by 2D Images: CueClash Benchmark Reveals 3D Spatial Reasoning Shortcomings
Meta Description
CueClash benchmark reveals that frontier vision-language models (Claude Opus 5.5, GPT-5.5, Gemini 3.6 Flash) rely on 2D image cues in 3D spatial reasoning, with shortcut gaps of 18-37 percentage points. Models achieve 99-100% accuracy when given ground coordinates, but top-down maps suffer from depth compression.
Article Content
Can vision-language models (VLMs) accurately judge three-dimensional spatial relationships? The new CueClash benchmark delivers a surprising answer: when 2D cues conflict with the true layout, frontier models systematically rely on image-plane cues rather than genuine 3D structure.
The Conflict Between 2D Cues and 3D Layout: How CueClash Separates the Two Judgments
In photographs, 2D plane cues (such as an object's vertical position in the frame) usually align with the 3D ground layout. CueClash renders scenes so that the 2D cues for five spatial relations are correct in exactly half of the statements and wrong in the other half, thereby quantifying models' reliance on 2D shortcuts.

The researchers define the "shortcut gap" as the accuracy when 2D cues are correct minus the accuracy when cues are wrong. For Claude Opus 5.5, GPT-5.5, and Gemini 3.6 Flash (low reasoning effort), this gap reaches +18 to +37 percentage points. Even after excluding two relations confounded by occlusion or boundary cases, the gap remains +7 to +29 percentage points. This shows that models rely heavily on image-plane cues rather than genuine depth understanding.
Shortcut learning is a common phenomenon in machine learning: models latch onto cues in training data that are statistically correlated with labels but are not essential to the task, performing well in-distribution and failing out-of-distribution. In vision-language models, 2D image-plane cues (such as objects lower in the frame being more likely closer to the viewer) align closely with true 3D layout in the vast majority of natural photographs, so models readily adopt them as a "shortcut" for judging spatial relations. CueClash's design deliberately breaks this correlation—by rendering scenes so that 2D cues are intentionally wrong in half the samples, it measures how much genuine spatial understanding models retain when "cues are unreliable."
Near-Perfect with Ground Coordinates: The Two Sides of Model Capability
When ground coordinates and relation definitions are provided directly, the three models answer with 99% to 100% accuracy. This shows that they possess conceptual knowledge of spatial relations; the problem lies in the conversion step from image to layout.

For scenes where objects pass exactly through an opening, pixel measurements using camera depth eliminate most of the shortcut gap. This further confirms that models are disturbed by depth foreshortening when extracting 2D cues—the top-down maps they draw preserve depth ordering but compress depth like a photograph.
Depth foreshortening is a geometric effect of perspective projection: when a camera captures the ground at a certain downward angle, distant objects not only appear smaller in the image but their ground distance along the line of sight is also compressed in pixels. This means pixel measurements alone cannot directly recover true ground distance; back-projection is required using parameters such as camera height and pitch angle. If a model lacks explicit modeling of camera intrinsics and pose, it will mistake the compressed pixel distance in the image for true distance, leading to systematic bias in ground-relation judgments. The fact that pixel measurements using camera depth eliminate most of the shortcut gap in this work precisely shows that correcting this geometric transformation is the key to the problem.
Depth Compression in Top-Down Maps: Differences Among Models
The top-down maps generated by the models reveal how they handle depth. GPT-5.5's map is almost a copy of the image, with depth information severely lost; Opus 5.5's and Gemini's maps recover about 0.6 of the depth foreshortening, but Gemini then uniformly shrinks all depths by a fixed factor. Opus 5.5 relies mainly on "fitting" to follow the image, and its map's degree of depth correction is comparable to that of humans observing real scenes.

These differences show that different models adopt different strategies when reconstructing 3D layout from 2D images, but none fully overcomes the depth compression effect of the image plane.
Bird's-eye-view (BEV) maps are representations that convert first-person-view images into a top-down layout, widely used in autonomous driving and robot navigation. Constructing a BEV requires inverse perspective mapping from the image plane to the ground plane, essentially back-projecting each pixel along the camera ray and intersecting it with the ground plane. This process highly depends on accurate estimation of camera extrinsics (height, pitch, yaw); if the camera parameters implicitly assumed by the model do not match reality, the BEV map will be stretched or compressed overall in the depth direction. The observation in this work that model maps "preserve depth ordering but compress depth" corresponds exactly to this kind of systematic geometric bias under parameter mismatch.
Failure on Real Photographs: Validation on CV-Bench
In the CV-Bench photo test, when reliable 2D cues are misleading, the accuracy of all three models also drops. This validates CueClash's findings: models are also misled by 2D shortcuts in real scenes.
For the two open-source models, layout information did reach the language model, but the map lost a large amount of information during transmission. This suggests that the interface between the vision encoder and the language model may also be a bottleneck.
The interface between the vision encoder and the language model refers to the projection layer (such as an MLP projector or Q-Former) that converts image features into token sequences consumable by the language model in multimodal models. This bottleneck is particularly prominent in spatial tasks: although the patch features output by the encoder retain local texture and semantics, after pooling or compression, fine-grained geometric and depth cues are often diluted, and the language model can only reason based on limited visual tokens. The phenomenon in open-source models that "layout information reached the language model but the map lost a large amount of information" suggests that even if the language model possesses conceptual spatial knowledge, information loss at the interface can degrade final judgments to reliance on 2D statistical shortcuts.
Implications for the Development of Multimodal AI
CueClash's research reveals a fundamental limitation of current VLMs in spatial reasoning: they tend to exploit statistical shortcuts in 2D images rather than construct genuine 3D scene understanding. This "image-plane bias" can lead to errors in tasks requiring precise spatial judgments.
Future improvement directions may include: designing better vision encoders to preserve depth information, introducing more samples in training where 2D cues conflict with 3D layout, and improving the model's conversion mechanism from image to layout. For applications that rely on spatial reasoning (such as robot navigation, autonomous driving, and augmented reality), this flaw deserves particular attention.
As model scale grows, will spatial reasoning capabilities improve naturally? CueClash's results show that even the most frontier models still exhibit significant shortcut reliance at low reasoning effort. This reminds us that genuine 3D understanding requires dedicated architectures and training strategies, not just larger parameter counts.
One-line Summary
CueClash benchmark reveals that frontier vision-language models rely on 2D image shortcuts rather than true 3D layout, with shortcut gaps of 18-37 percentage points.
Paragraph Summary
The new CueClash benchmark renders scenes so that 2D plane cues for five spatial relations are correct in exactly half of the statements and wrong in the other half, thereby separating models' reliance on 2D shortcuts from true 3D layout. Results show that Claude Opus 5.5, GPT-5.5, and Gemini 3.6 Flash suffer significant accuracy drops when 2D cues are wrong, with shortcut gaps of 18-37 percentage points, and 7-29 percentage points even after excluding easily confused relations. However, when ground coordinates are provided directly, the three achieve 99%-100% accuracy, indicating complete conceptual knowledge and a bottleneck in the image-to-layout conversion. Model-generated top-down maps preserve depth ordering but compress depth: GPT-5.5 nearly copies the image, while Opus 5.5 and Gemini can partially correct. Real photographs on CV-Bench confirm the same misleading 2D cue effect.
Related articles

RePlan-VLA: Progress-Aware Replanning Breakthrough for Long-Horizon Robotic Manipulation
The RePlan-VLA framework converts VLA models' short-horizon capabilities into reliable long-horizon execution through progress-aware adaptive replanning, improving success rates by 21.9% and 10.4% on RoboCasa365 and CALVIN, and validating real-world deployment on a Franka Research 3 robot.

Geometric Bridge Flow Matching: A New Breakthrough in Blind Rigid Motion Correction
Geometric bridge flow matching enables blind rigid motion correction in head CBCT, achieving 37.1 dB PSNR and 0.27 mm RPE on simulated data in ~9.1 minutes, outperforming existing methods.

Google Releases Gemini 4 Hydrogen: 2.5x Faster Without Sacrificing Intelligence
Google releases the lightweight Gemini 4 Hydrogen: 2.5x faster task completion, AA Intelligence Index of 54 beating its flagship sibling, 7.3s single-step on Agent Arena faster than Claude and GPT rivals, four thinking levels with pay-as-you-need pricing, now available across AI Studio and more.