SpaceCast-Bench
A benchmark for evaluating the predictive spatial reasoning ability of vision-language models. It proposes the observe-transform-infer framework and contains 3,862 questions drawn from 182 real-world scenes, covering 16 task types across three levels.
Timeline (last 90 days)
SpaceCast-Bench is the first benchmark that directly and diagnostically evaluates predictive spatial reasoning ability.
SpaceCast-Bench proposes the observe-transform-infer framework, which requires models to observe real scenes, face spatial interventions, and answer questions about the transformed scene, which is not directly shown in the input.
SpaceCast-Bench contains 3,862 questions from 182 real scenes, covering 16 task types, divided into three levels: static perception, local prediction, and global prediction.
Among the 21 models evaluated on SpaceCast-Bench, the strongest model reaches only 58.0% accuracy, while human performance is 87.2%.
Models specifically optimized for spatial tasks perform almost close to random guessing on SpaceCast-Bench.
Controlled experiments found that bridge views are crucial for integrating scattered observations, and removing them causes a significant drop in model performance.
On the Qwen3-VL-4B model, fine-tuning with procedurally generated data raises its accuracy on SpaceCast-Bench from 34.0% to 65.7%, and achieves a macro-average gain across six out-of-domain benchmarks.
Even after fine-tuning to reach 65.7% accuracy, there is still a considerable gap from the human level of 87.2%.
SpaceCast-Bench can be applied to scenarios such as robotic manipulation, autonomous driving, and AR/VR interaction, providing a quantifiable diagnostic tool.
All Facts (9)
SpaceCast-Bench is the first benchmark that directly and diagnostically evaluates predictive spatial reasoning ability.
50%UnverifiedSpaceCast-Bench proposes the observe-transform-infer framework, which requires models to observe real scenes, face spatial interventions, and answer questions about the transformed scene, which is not directly shown in the input.
50%UnverifiedSpaceCast-Bench contains 3,862 questions from 182 real scenes, covering 16 task types, divided into three levels: static perception, local prediction, and global prediction.
50%UnverifiedAmong the 21 models evaluated on SpaceCast-Bench, the strongest model reaches only 58.0% accuracy, while human performance is 87.2%.
50%UnverifiedModels specifically optimized for spatial tasks perform almost close to random guessing on SpaceCast-Bench.
50%UnverifiedControlled experiments found that bridge views are crucial for integrating scattered observations, and removing them causes a significant drop in model performance.
50%UnverifiedOn the Qwen3-VL-4B model, fine-tuning with procedurally generated data raises its accuracy on SpaceCast-Bench from 34.0% to 65.7%, and achieves a macro-average gain across six out-of-domain benchmarks.
50%UnverifiedEven after fine-tuning to reach 65.7% accuracy, there is still a considerable gap from the human level of 87.2%.
50%UnverifiedSpaceCast-Bench can be applied to scenarios such as robotic manipulation, autonomous driving, and AR/VR interaction, providing a quantifiable diagnostic tool.
50%