VLM Spatial Reasoning Benchmark: Which Model Wins on 7×7 Grid Puzzles?

Visual language models (VLMs) excel at multimodal understanding and image captioning — but how well can they actually perform step-by-step visual-spatial verification? One developer built a synthetic benchmark called VCIS-Bench (Visual Constraint & Instruction Simulation) that puts several leading VLMs through the same 7×7 grid puzzle test, measuring their spatial reasoning capabilities head-to-head.

Test Design: A Rigorous Grid Puzzle

At the heart of VCIS-Bench is a strictly structured 7×7 grid. Each puzzle contains fixed elements: an agent (A), a key (K), a locked door (D), a goal (G), and walls (#). The rules themselves aren't complicated, but they place clear demands on the model's state-tracking ability — the agent can only move in cardinal directions (up, down, left, right), cannot pass through walls, and must collect the key on the left side of a central barrier before unlocking the door and reaching the goal on the right.

The elegance of this design is that it breaks down an abstract question — "can this model reason?" — into a concrete, quantifiable task. The model doesn't just need to "read" the grid; it must mentally simulate a causal chain of actions — precisely the weak spot for many multimodal models.

The benchmark consists of three core tasks:

• Task 1 | Overall Plan Validation: Given an action sequence, determine whether the plan is VALID or INVALID.
• Task 2 | Spatial Location Verification: Using labeled row (R1–R7) and column (C1–C7) headers, verify object coordinates to assess spatial perception.
• Task 3 | Step-Level Error Diagnosis: Identify the exact step index (1-based) where a rule is first violated in an action plan — such as hitting a wall or passing through a door without a key.

The three tasks escalate in difficulty, forming a capability gradient from "binary judgment" to "precise error localization."

From a cognitive science perspective, the core capability tested by these grid puzzles is known as mental simulation — the ability to reason through state transitions "in the mind" without physically executing actions. For humans, this relies on working memory and spatial representation. For language models, it requires maintaining an implicit "world state" during token generation, updating coordinates, collision detection, and conditions (key possession) at every step. Researchers distinguish this from pure visual perception (grounding): perception is "identifying what is in the image and where," while simulation is "given this action, what is the next state?" Most existing VLM architectures are optimized for the former; the latter lacks sufficient supervision in training data and objective functions — which is the fundamental reason benchmark results show clear performance stratification.

Competing Models: A Three-Tier Showdown

The benchmark selected three models with distinctly different profiles, spanning top closed-source to open-weight options, to simultaneously measure accuracy, latency, and cost efficiency:

• Claude Sonnet 5.5 (Anthropic): Serves as the reference standard for current visual reasoning.
• Gemini 3.7 Flash (Google): Google's lightweight multimodal model optimized for fast response.
• Gemma 4 26B A4B (Google): A representative open-weight visual-language architecture.

This combination clearly illustrates the performance gaps between top-tier closed-source frontier models, lightweight API endpoints, and open-source models — rather than just producing a simple leaderboard.

Results: Who Held Up Under Step-by-Step Simulation?

Overall, the performance gaps between models were quite pronounced.

Task	Claude Sonnet 5.5	Gemini 3.7 Flash	Gemma 4 26B A4B
Task 1: Overall Plan Validation	100.0%	100.0%	83.3%
Task 2: Spatial Location Verification	100.0%	100.0%	96.7%
Task 3: Step-Level Error Diagnosis	100.0%	92.0%	76.0%
Overall Score	1.00	0.97	0.85

(Data evaluated on October 11, 2026)

Step-Level Simulation Is the Biggest Bottleneck

The most striking finding is this: what separates the models isn't "seeing" — it's "computing." Claude Sonnet 5.5 achieved a perfect 100% score across all three tasks. But other models began to crack once they entered Task 3, which requires precise state tracking. Gemini 3.7 Flash scored perfectly on the binary verification tasks (Tasks 1 and 2), but its accuracy dropped to 92.0% when asked to pinpoint exactly which step caused the failure.

This suggests that judging "right or wrong" is relatively straightforward, but tracking and localizing "which step went wrong" across multi-step state transitions is a completely different order of challenge for a model's internal state management.

From a technical standpoint, binary verification tasks (Tasks 1 and 2) allow models to take heuristic shortcuts — for instance, checking the overall plausibility of an action sequence or directly aligning visual markers with coordinate labels, without strictly executing step by step. Task 3, however, requires outputting a specific failure step index, meaning the model must update the agent's position, detect collisions, and evaluate key possession from the very first step — and any state error compounds forward, causing the final index to be off. This error accumulation effect amplifies over longer action sequences: even if single-step accuracy drops by only a few percentage points, overall localization precision across chained steps can degrade significantly. This mirrors the notoriously tricky "off-by-one error" in programming, and neatly explains why models with high spatial perception scores can still stumble on step-level diagnosis.

Spatial Grounding vs. Open-Source Execution

Gemma 4 26B A4B's performance offers an interesting perspective. It scored 96.7% on Task 2's spatial coordinate verification, demonstrating that open-weight models have become quite mature at visual grounding. However, its score dropped to 76.0% on Task 3's step-level execution, precisely due to accumulated tracking errors across multi-step state transitions. In short, open-source models "know where things are," but still show a meaningful gap when "simulating the outcome of a sequence of actions."

The Cost-Efficiency Sweet Spot

Shifting the lens from pure accuracy to cost efficiency yields more practically useful insights. Looking at the author's Pareto plot of "score vs. total cost," Gemini 3.7 Flash offers a compelling tradeoff — achieving a near-perfect score of 0.97 at roughly half the total cost and execution latency of Claude Sonnet 5.5. For budget-sensitive applications that still require strong spatial reasoning, this "cost-efficiency sweet spot" is worth serious consideration.

Pareto optimality is a common analytical framework in engineering decisions: in multi-objective optimization, a solution is Pareto optimal if no other solution can improve one metric without degrading another. In the two-dimensional space of "accuracy vs. cost," Gemini 3.7 Flash sits near the Pareto frontier — it's neither the most accurate nor the cheapest, but it achieves the highest accuracy at equivalent cost, or the lowest cost at equivalent accuracy, making it the rational choice for cost-sensitive deployments. This type of analysis is more practically valuable than a simple leaderboard, since real-world deployments almost never operate under "unlimited budget, maximize accuracy only" conditions — most engineering decisions are fundamentally multi-objective tradeoffs.

Implications for Real-World Applications

Despite its modest scale, this benchmark highlights an aspect of VLM deployment that's easy to overlook: many models are already quite capable at "recognition" and "description," but remain unreliable on tasks requiring continuous, deterministic state tracking. For scenarios like robot navigation, game AI, and process automation that depend on precise multi-step reasoning, selecting a model based solely on spatial perception scores is insufficient — step-level diagnostic capability is the real litmus test.

The author has published the task code and interactive leaderboard results on Kaggle Benchmarks (VCIS Benchmark), where interested developers can reproduce or extend the test framework. For teams looking to evaluate their models' spatial reasoning abilities, this is a lightweight yet targeted starting point.