Grid-Bench
An executable benchmark for evaluating the discrete spatial state prediction capability of large language models. It simulates discrete worlds using two- to four-dimensional grids, applies sequences of spatial operations, and requires the model to predict the final grid state, with a reference executor verifying the output cell by cell.
Timeline (last 90 days)
The research team released an executable benchmark named Grid-Bench for evaluating LLMs' ability to predict the final state after multi-step operations in a discrete world.
Grid-Bench uses two-dimensional, three-dimensional, or four-dimensional grids to represent a discrete world and successively applies spatial operations such as moving objects, transforming regions, and modifying the surrounding environment to the world.
Grid-Bench actually runs operations through a reference executor, and every cell value output by the model can be checked item by item, making it a reproducible, auditable, precise computation task.
The researchers used Grid-Bench to evaluate six LLMs and found that as the operation history grows longer, the accuracy of predicting the complete state drops sharply.
Models that perform similarly on single-event tasks diverge significantly from one another once multi-step history is involved, and single-step performance can hardly predict multi-step performance.
By recording cell-by-cell correctness, Grid-Bench provides a partial-credit perspective, upgrading evaluation from right/wrong to where it went wrong and how much it got wrong.
LLMs still have clear shortcomings in predicting multi-step changes in discrete worlds, and their ability to track spatial operations and infer states gradually decays over long chains.
All Facts (7)
The researchers used Grid-Bench to evaluate six LLMs and found that as the operation history grows longer, the accuracy of predicting the complete state drops sharply.
50%UnverifiedGrid-Bench uses two-dimensional, three-dimensional, or four-dimensional grids to represent a discrete world and successively applies spatial operations such as moving objects, transforming regions, and modifying the surrounding environment to the world.
50%UnverifiedGrid-Bench actually runs operations through a reference executor, and every cell value output by the model can be checked item by item, making it a reproducible, auditable, precise computation task.
50%UnverifiedLLMs still have clear shortcomings in predicting multi-step changes in discrete worlds, and their ability to track spatial operations and infer states gradually decays over long chains.
50%UnverifiedModels that perform similarly on single-event tasks diverge significantly from one another once multi-step history is involved, and single-step performance can hardly predict multi-step performance.
50%UnverifiedBy recording cell-by-cell correctness, Grid-Bench provides a partial-credit perspective, upgrading evaluation from right/wrong to where it went wrong and how much it got wrong.
50%UnverifiedThe research team released an executable benchmark named Grid-Bench for evaluating LLMs' ability to predict the final state after multi-step operations in a discrete world.
50%