DK-VLM: Treating Geometry as Position to Crack VLM Spatial Reasoning

The Bottleneck of Spatial Reasoning: Geometry Struggles to Integrate with Semantic Representations

Vision-language models have become quite mature at recognizing objects and describing scenes, but performance drops noticeably once cross-view spatial judgments are involved—questions like "is the cup to the left of or behind the table" or "after walking to the other side of the room, which direction is the TV in." The core difficulty is that the model needs to understand both semantic content and 3D geometric relationships, and the representations of these two are inherently different.

According to the paper, a common approach is to directly fuse 3D reconstruction features into visual tokens, but the authors point out that with limited spatial supervision signals, aligning these geometric features with the semantic representations already present in pre-trained models remains an unsolved problem. DK-VLM proposes a different angle: rather than squeezing geometry into tokens as "extra features," let geometry participate directly in attention computation as "positional information."

Vision-language models (VLMs) typically consist of a visual encoder (such as ViT) and a language model. The visual encoder divides images into patches and converts them into token sequences, which are then fed into the Transformer along with text tokens. This architecture is naturally good at handling semantic tasks like "what is in the image," but lacks the ability to model "3D spatial relationships between objects"—because the image itself is a 2D projection, and information such as depth, occlusion, and absolute scale is lost during projection.

3D reconstruction features refer to representations such as point clouds, voxels, or depth maps inferred from multi-view images or monocular depth estimation networks. These features carry true geometric information, but their numerical distributions, dimensions, and semantic meanings differ greatly from pre-trained visual tokens, and direct concatenation often leaves the model "unable to understand" them—this is the root of the alignment difficulty the paper emphasizes.

Dual-Kernel Attention: Semantics and Geometry Each Play Their Part

The core design of DK-VLM is Dual-Kernel Attention. Traditional Transformer attention is determined by the token's input position, while DK-VLM adds a geometric kernel on top of this: using estimated 3D point coordinates and camera poses, it directly modulates attention weights through point displacement and relative transformations between viewpoints.

The division of labor between the two kernels is clear. The semantic kernel preserves the pre-trained model's native attention and positional encoding, ensuring the model does not lose already-learned semantic capabilities due to the introduction of geometric information. The geometric kernel handles spatial relationships—it does not rely on fused reconstruction features, but instead uses raw 3D points and camera parameters to compute spatial associations between tokens. The outputs of the two kernels are fused through a learnable gating mechanism, allowing the model to decide for itself when to rely more on semantics and when to rely more on geometry.

This "geometry as position" design bypasses the feature alignment problem. Geometric information no longer needs to be "translated" into a format the semantic tokens can understand, but instead enters the attention mechanism directly in the role of a positional signal.

Attention mechanism centers on the dot product of query and key, which determines which other tokens each token should attend to. In standard Transformers, this weight is determined only by the token's content and positional encoding, unrelated to the scene's true 3D structure. DK-VLM's approach adds a bias term computed by the geometric kernel on top of the dot product result, essentially telling the model "from a 3D perspective, these two tokens should already be closer."

Gating mechanism is a learnable weighted fusion method, typically of the form g·A_sem + (1-g)·A_geo, where g is dynamically generated from input features through a small network. This way, the model can lower the geometric kernel's weight for purely semantic tasks (such as object recognition), and increase geometric kernel involvement when spatial judgments are needed, avoiding interference from geometric signals on semantic capabilities.

Reference-Frame Composition: Teaching the Model to Update Reference Frames

A key capability in spatial reasoning is reference frame updating. When you walk from one side of a room to the other, the relative orientation of the same object changes, and the model needs to track this change and update its spatial judgments.

DK-VLM introduces Reference-Frame Composition (RC) supervision to address this. Specifically, it derives step-by-step turning and heading targets from verified single-step annotations, using these intermediate signals to train the model to update its reference frame across consecutive judgments. This means the model must not only know "where the object is now," but also learn "where it would be if I turned in a certain direction."

The value of this supervision approach is that it decomposes multi-step spatial reasoning into a trainable step-by-step updating process, rather than requiring the model to intuit the rules of reference frame transformation directly from single-step annotations.

Reference frame is a foundational concept in spatial cognition, referring to the coordinate system used when describing an object's orientation. The same object has completely different orientations under a "viewer-as-origin" reference frame versus a "room-as-origin" reference frame. Humans can easily switch between them through mental rotation, but models need explicit training to master this.

Single-step annotation refers to supervision signals that only provide a single judgment such as "from the current viewpoint, A is to the left of B," without including the viewpoint transformation process. DK-VLM reverse-engineers turning angles and heading changes from these single-step annotations to construct pseudo-labels for intermediate steps, effectively providing the model with supervision on "how to rotate from viewpoint A to viewpoint B"—this is precisely what distinguishes RC supervision from ordinary multi-step question-answering training.

Benchmark Performance

Across six spatial benchmarks, DK-VLM-8B achieved the highest average score. It scored 74.33 on VSI-Bench and 58.28 on ReVSI-32, both the highest total scores. The addition of RC supervision further improved multi-step spatial reasoning performance.

Notably, the full model retains general video understanding capabilities while enhancing spatial abilities. This is not common in spatial reasoning research—many methods sacrifice the model's generalization on other tasks to improve spatial judgment accuracy. Through the dual-kernel separation design, DK-VLM keeps the geometry processing module and semantic understanding module relatively independent, thereby reducing mutual interference between capabilities.

Why This Approach Is Worth Attention

DK-VLM's insight lies in its redefinition of the problem. While most attempts try to make geometric features "better integrate" with semantic representations, DK-VLM chooses to let geometry enter the model through a completely different channel—through positional modulation of the attention mechanism, rather than feature fusion.

The potential advantages of this approach are modularity and scalability. The geometric kernel relies on 3D points and camera poses, which can be directly obtained in scenarios such as robot navigation and AR/VR, without needing additional feature extraction networks. Meanwhile, the preservation of the semantic kernel means the model can reuse existing pre-trained weights, reducing training costs.

Of course, the effectiveness of this method depends on the accuracy of 3D point estimation and camera poses. In real-world scenarios, these inputs often contain noise, and the robustness of the geometric kernel still needs further validation. The paper's current results are based on benchmarks, and performance in open environments remains to be seen.

Project homepage: https://sumuru789.github.io/DK-VLM-website/

Camera pose typically consists of a rotation matrix R and a translation vector t, describing the camera's position and orientation in the world coordinate system. It is a key parameter connecting image pixels with 3D space, and in SLAM, robot navigation, and AR/VR it can be directly obtained through IMU, visual odometry, etc., with accuracy far higher than depth estimated from a single image. DK-VLM's geometric kernel leverages this prior, converting pose information into relative transformations between tokens, thereby giving attention weights spatial consistency.

Robustness refers to a model's ability to maintain performance when inputs contain noise, outliers, or distribution shifts. 3D point estimation errors and pose drift will directly propagate to the geometric kernel's attention bias, potentially causing the model to attend to incorrect tokens in critical regions. The paper has not systematically evaluated performance under noisy conditions, which is an important direction to verify before this method can move toward real-world deployment.