SpatialASM: Teaching Vision-Language Models to Actively Imagine Space

When Machines Need to Think from a Different Angle

When tackling complex spatial reasoning problems, humans rely on an almost instinctive strategy: mentally shift to a task-relevant viewpoint and reconstruct the spatial layout from that perspective. Imagine being asked, "Looking from the other side of the room, is the table to the left or right of the chair?" Most people will instinctively "walk" to that position in their mind before answering.

Researchers call this ability Active Spatial Mentalization (ASM). It is an intrinsic and critical component of visual intelligence — yet it remains underdeveloped in today's large vision-language models (LVLMs). These models can recognize objects and describe images, but they struggle to actively engage the "imagine it from another angle" reasoning mechanism.

Addressing this gap, a new study called SpatialASM proposes a scalable framework that aims to internalize ASM capability into vision-language models during training, enabling them to proactively plan viewpoints and simulate scenes — much like humans do.

Large Vision-Language Models (LVLMs) are multimodal systems that combine visual encoders with large language models. Representative examples include GPT-4V, LLaVA, and InternVL. These models convert images into visual tokens, which are fed alongside text tokens into the language model to enable joint visual-linguistic understanding. While LVLMs excel at tasks like image captioning and visual question answering, their reasoning is fundamentally rooted in pattern matching over visual features — not active modeling of 3D spatial relationships. This causes them to systematically underperform humans on tasks requiring "imagining the scene from a third-party perspective." Models can typically answer "what is visible in the image" but struggle to infer "what would it look like from a different position." Closing this capability gap is precisely what SpatialASM targets.

What SpatialASM Does

The core idea behind SpatialASM is training-time spatial simulation. Rather than relying on external tools at inference time, the framework builds high-quality supervision signals so that models develop active spatial imagination habits during learning itself.

A key component of the framework is the allocentric novel-view generator. It anchors hypothetical viewpoints to real visual evidence, ensuring that the scenes a model imagines are grounded in the image rather than fabricated from nothing.

Building on this generator, SpatialASM produces two complementary types of supervision data:

Explicit ASM Reasoning Trajectories

The first type is explicit ASM trajectories. This data teaches the model when and how to adaptively invoke viewpoint planning and imagination during spatial reasoning. Rather than simply providing the final answer, it demonstrates the complete thought process — when a perspective shift is needed, which perspective to adopt, and what the imagined scene looks like. This essentially provides specialized Chain-of-Thought (CoT) supervision for spatial reasoning.

Chain-of-Thought (CoT) is a training and prompting technique introduced by Wei et al. in 2022 that encourages models to show intermediate reasoning steps before arriving at a final answer. Research shows that CoT supervision significantly improves model performance on mathematics, logic, and commonsense reasoning tasks by decomposing complex problems into verifiable sub-steps. However, CoT effectiveness heavily depends on high-quality "reasoning demonstration" data — requiring annotators to write out each thinking step, which is extremely costly. Spatial reasoning CoT is particularly challenging: annotators must not only provide correct answers but also explicitly articulate perspective-switching steps like "first imagine standing here, then describe the layout as seen from there." Such data is virtually absent in existing datasets. By synthesizing explicit ASM trajectories, SpatialASM effectively auto-generates CoT training data for the specific domain of spatial reasoning.

Automatically Constructed Perspective-Taking Tasks

The second type is automatically constructed perspective-taking tasks. These tasks further reinforce the model's ability to imagine scenes from specific viewpoints, making it more proficient at handling questions like "what would it look like from a hypothetical position?" Since the tasks are constructed automatically, the entire pipeline is highly scalable.

Results: Validation Across Seven Benchmarks

The research team evaluated SpatialASM on seven benchmark tests, spanning tasks from perceptual spatial understanding to more challenging multi-image, cross-viewpoint, and perspective-transformation reasoning.

Results show that SpatialASM delivers spatial reasoning improvements across four different LVLM backbone models, while preserving the models' original general visual capabilities — a particularly important finding, since targeted fine-tuning often comes at the cost of generalization. The experimental data demonstrates average accuracy improvements across all seven spatial benchmarks.

This consistent improvement across different backbones and tasks indicates that ASM, as a transferable capability, can indeed be systematically instilled into diverse vision-language models — rather than being an incidental benefit specific to any one architecture.

Why This Work Matters

A practical bottleneck facing spatial reasoning research today is the scarcity of spatial chain-of-thought supervision data. Teaching models to actively explore viewpoints and simulate mental scenes requires abundant high-quality demonstrations of "how to think" — data that is extremely difficult to annotate manually.

SpatialASM addresses this through synthetic ASM trajectories, offering a scalable path to alleviate this data scarcity. It transforms what was once a labor-intensive process of "demonstrating reasoning" into automatically generated training assets — a methodological contribution with broad implications for future research in this field.

From a broader perspective, this work touches on a fundamental question: true visual intelligence is not just about "seeing clearly" — it requires "thinking spatially." The ability to actively shift perspectives and reconstruct spatial relationships in one's mind is one of the core advantages of human spatial cognition. SpatialASM represents a meaningful step toward equipping machines with this active imaginative capacity.

Allocentric and egocentric perspectives are a foundational pair of concepts in spatial cognition research. An egocentric perspective uses the observer as the reference frame ("to my left," "in front of me"), while an allocentric perspective uses fixed environmental landmarks or third-party positions as the reference frame ("the table is on the north side of the room," "the chair is to the right of the table as seen from the doorway"). Humans fluidly switch between both perspectives — a capability closely linked to spatial encoding mechanisms in the hippocampus and parietal cortex. Most current visual datasets and models are egocentric by nature, describing scenes from the photographer's viewpoint, which makes it difficult for models to perform tasks requiring allocentric reasoning. The "allocentric novel-view generator" in SpatialASM is specifically designed to address this bias — it forces the model to learn to reconstruct spatial relationships from any hypothetical position as the origin, rather than relying solely on the original camera perspective.

Summary

SpatialASM abstracts the human behavior of "perspective-shifting imagination" in spatial problem-solving into a trainable capability. By combining novel-view generation, explicit reasoning trajectories, and automated perspective-taking tasks, it systematically improves the spatial reasoning performance of vision-language models without compromising their general-purpose capabilities. For researchers and engineers focused on multimodal reasoning and visual intelligence, it offers both a concrete technical solution and a viable direction for using synthetic data to address the broader challenge of scarce CoT supervision data.