VCIS-Bench
A synthetic benchmark designed to evaluate the spatial reasoning capabilities of vision-language models (VLMs). Its core design employs 7×7 grid puzzles and encompasses three tasks: overall solution verification, spatial localization verification, and step-level error diagnosis. The benchmark is intended to assess a model's capacity for mental simulation and state tracking.
Timeline (last 90 days)
VCIS-Bench (Visual Constraint & Instruction Simulation) is a synthetic benchmark built by a developer that uses 7×7 grid puzzles to evaluate the spatial reasoning capabilities of VLMs.
The 7×7 grid puzzles in VCIS-Bench contain fixed elements including an agent (A), a key (K), a locked door (D), a goal (G), and walls (#); the agent can only move in cardinal directions, cannot pass through walls, and must collect the key before unlocking the door to reach the goal.
VCIS-Bench tests revolve around three core tasks: overall solution verification (judging VALID/INVALID), spatial localization verification (verifying object coordinates), and step-level error diagnosis (identifying the step number where the first rule violation occurs).
Claude Sonnet 5.5 achieved a perfect score of 100% on all three tasks in VCIS-Bench, with a total score of 1.00.
The evaluation date for this benchmark is October 11, 2026.
The test conclusions show that what truly differentiates models is not visual perception but step-by-step simulation capability — judging right from wrong is relatively easy, but locating which step went wrong in a multi-step state transition is a higher-order challenge for a model's internal state management.
The authors have published the VCIS-Bench task code and interactive leaderboard run results on the Kaggle Benchmarks platform.
All Facts (7)
VCIS-Bench (Visual Constraint & Instruction Simulation) is a synthetic benchmark built by a developer that uses 7×7 grid puzzles to evaluate the spatial reasoning capabilities of VLMs.
50%UnverifiedThe 7×7 grid puzzles in VCIS-Bench contain fixed elements including an agent (A), a key (K), a locked door (D), a goal (G), and walls (#); the agent can only move in cardinal directions, cannot pass through walls, and must collect the key before unlocking the door to reach the goal.
50%UnverifiedVCIS-Bench tests revolve around three core tasks: overall solution verification (judging VALID/INVALID), spatial localization verification (verifying object coordinates), and step-level error diagnosis (identifying the step number where the first rule violation occurs).
50%UnverifiedClaude Sonnet 5.5 achieved a perfect score of 100% on all three tasks in VCIS-Bench, with a total score of 1.00.
50%UnverifiedThe evaluation date for this benchmark is October 11, 2026.
50%UnverifiedThe test conclusions show that what truly differentiates models is not visual perception but step-by-step simulation capability — judging right from wrong is relatively easy, but locating which step went wrong in a multi-step state transition is a higher-order challenge for a model's internal state management.
50%UnverifiedThe authors have published the VCIS-Bench task code and interactive leaderboard run results on the Kaggle Benchmarks platform.
50%