Grid-Bench: Can LLMs Predict Changes in a Discrete World?

Large language models perform impressively on text generation and reasoning tasks, but when the task becomes "predict what a discrete world looks like after a series of operations," can they maintain the same level of performance? A new paper offers a fairly nuanced answer to this question through an executable benchmark called Grid-Bench.

What Is Grid-Bench

The core setup of Grid-Bench is not complicated: a discrete world is represented as a 2D, 3D, or 4D grid, and then a series of spatial operations is applied to this world in sequence—moving objects, transforming regions, modifying surrounding environments, and so on. Each operation acts on the configuration left by the previous step. The initial grid, explicit rules, and the operation history given in text form are all provided to the model, which is asked to predict the complete final grid.

What gives this benchmark real weight is its "executable" property. The researchers actually run these operations using a reference executor, so every cell value in the model's output can be checked one by one. This means one thing: the model isn't just being asked "what do you think it will become"—it's being precisely tested on "did you calculate it correctly?" This design transforms spatial operation prediction from a vague perceptual task into a quantifiable, reproducible, precise computation task.

Executable Benchmark is the key design decision in this paper that determines Grid-Bench's credibility. In traditional benchmarks, model outputs are typically compared against human-annotated "ground truth," but the annotations themselves may contain ambiguity, errors, or incomplete coverage; an executable benchmark instead requires researchers to provide a Reference Executor that can actually run the task, executing operations on given inputs and producing deterministic outputs, against which model predictions are then compared item by item. Since the executor is code, its behavior is fully reproducible and auditable—there's no issue of "different annotators understanding things differently." This paradigm is already relatively mature in areas like code generation (where unit tests serve as executors), SQL generation, and mathematical solving, but it remains uncommon in spatial reasoning tasks. Grid-Bench essentially transplants the idea of "run it and see if it's correct" into discrete world prediction.

The Longer the History, the More Models "Break Down"

The researchers evaluated six LLMs using Grid-Bench, and the results revealed a clear trend: as the operation history grows longer, the accuracy of complete state prediction drops sharply.

This finding itself is not surprising—the longer the sequence, the more opportunities for error accumulation, and any mistake at an intermediate step contaminates all subsequent steps. But the study also revealed a more subtle contrast: models that perform similarly on single-event tasks diverge significantly once they enter multi-step histories. In other words, "single-step performance" can hardly predict "multi-step performance."

This has direct implications for evaluation practices. If models are ranked using only short sequences or single-operation tasks, the conclusions drawn are likely to fail in real-world long-chain spatial reasoning scenarios.

Same Accuracy, Completely Different Error Distributions

The paper points out an easily overlooked issue: two models may be comparable in exact match accuracy, but on shared test instances, the number of cells they get wrong can differ dramatically.

This suggests that a single overall accuracy metric masks differences in "how badly" models fail. One model might fail on only a few instances but get them completely wrong each time; another might fail on more instances but only miss one or two cells each time. For scenarios that need partially correct results—such as assisted planning or state estimation—the latter might actually be more useful. By recording the correctness of each individual cell, Grid-Bench makes this difference measurable.

Exact Match Accuracy means that the model's entire output grid must be cell-by-cell identical to the reference grid for the instance to pass; even a single wrong cell counts as failure. This metric is strict but coarse: it treats "off by one cell" and "completely wrong" as equivalent, and it aggregates errors within an instance into a binary 0 or 1 result. When sample sizes are limited, several different models might land on the same accuracy because "the instances they got wrong happen to be different ones," yet their per-cell error counts, whether errors concentrate in specific regions or specific operation types, can differ vastly. By recording per-cell correctness, Grid-Bench effectively provides a partial credit perspective beyond exact match, upgrading evaluation from "right/wrong" to "where did it go wrong and by how much"—which is especially valuable for scenarios like assisted planning that require intermediate states.

"Partial Recovery" Ability Under Long Histories

Even in long-history tasks, the performance of some models is noteworthy. The study shows that some models can capture a considerable number of correct changes in long sequences, performing significantly above the "copy the initial state" baseline.

This means models aren't completely "guessing blindly." They are indeed performing some kind of spatial operation tracking and state inference—it's just that this ability gradually decays over long chains. This finding has reference value for understanding LLM internal mechanisms: models seem to possess a certain "discrete state transition" capability, but it's far from sufficient for stably completing multi-step precise inference.

Copy-Initial-State Baseline serves as an important reference anchor in long-history tasks. Since after multiple operations the grid often changes only locally, with the overall layout remaining similar to the initial state, a trivial strategy of "do nothing and directly output the initial grid" can achieve a non-trivial score on cell-level accuracy. If a model's performance merely matches this baseline, it means it isn't truly tracking state transitions—it's just "getting lucky" due to the statistical regularity that "most cells happen to be unchanged." Researchers therefore need models to clearly exceed this baseline to prove they are actually doing state inference rather than static copying. This also explains why Grid-Bench reports cell-level metrics: looking only at exact match, the trivial strategy scores very low; but looking at cell-level accuracy, the baseline becomes threatening, which in turn better reveals whether the model is truly "computing."

Why This Benchmark Is Worth Paying Attention To

Spatial instructions, state transformations, multi-step operations—these tasks appear everywhere in robot planning, game state inference, code execution simulation, and other scenarios. Grid-Bench's value lies in abstracting these tasks into a clean, verifiable testbed.

All predicted values can be checked by the reference executor, making it possible to compare across different spatial operation types, different history lengths, and different workloads. For researchers looking to evaluate or improve LLM spatial reasoning capabilities, this is a more fine-grained measurement tool than "Q&A accuracy."

Current results show that LLMs still have clear shortcomings in multi-step change prediction in discrete worlds, and Grid-Bench has precisely exposed these shortcomings under measurable light.

One-line Summary

The executable benchmark Grid-Bench reveals that LLMs degrade sharply on multi-step spatial operation prediction in discrete worlds as history grows longer.

Paragraph Summary

The paper proposes an executable benchmark, Grid-Bench, which represents discrete worlds as 2D to 4D grids, sequentially applies spatial operations like moves and transformations, and asks models to predict the final grid from the initial grid, rules, and textual operation history. Because a reference executor actually runs the operations, every cell value can be checked individually. Evaluation of six LLMs found that complete-state accuracy drops sharply as operation history grows; models with similar single-step performance diverge significantly on multi-step tasks; and models with identical exact match accuracy can differ dramatically in the number of wrong cells, showing that a single metric masks differences in error distribution. Even under long histories, some models can still recover many correct changes, clearly exceeding the copy-initial-state baseline, indicating a certain ability to track state transitions—but still insufficient for stably completing multi-step precise inference.