Nonobench: An Open-Source Benchmark Testing 49 LLMs on Nonogram Puzzles

Testing Spatial Reasoning in LLMs with Nonogram Puzzles

How good are large language models at structured logical reasoning? The open-source project Nonobench offers a fascinating angle: pitting 49 LLMs against nonogram puzzles (also known as picross). These puzzles require players to determine which cells in a grid should be filled based on numerical clues provided for each row and column, ultimately forming a pattern. They test both logical deduction and a model's ability to maintain "spatial memory" across long grid sequences.

The testing rules are strict: each model receives the row and column clues exactly once and must return the complete grid in a single response. No external tools are allowed, and each puzzle gets only one attempt. This "bare exam" setup strips away any tool-augmented advantages, directly measuring the model's raw reasoning capability.

Methodology: From 5×5 to 20×20

The benchmark is divided into two difficulty tiers. The standard mode contains 30 puzzles with grid sizes ranging from 5×5 up to 15×15, drawn from the Nonograms dataset published by Moyà-Alcover (CC BY 4.0 license).

The hard mode switches to ten randomly generated 20×20 grids, each verified to have a unique solution. Five of these cannot be solved using line logic alone — they require deeper cross-row-and-column joint reasoning — which is precisely the threshold that distinguishes models by reasoning depth. Notably, the grids are randomly filled rather than depicting real images, to prevent models from guessing by visual priors: if the answer were a cat or a tree, a model might guess correctly based on image familiarity, while random fills eliminate that shortcut.

A total of 130 variants were tested, covering different levels of reasoning effort, routed through OpenRouter and pinned to each lab's official endpoints wherever possible to ensure comparability.

Line logic is the most fundamental reasoning strategy in nonogram puzzles: determining which cells must be filled or empty based solely on a single row or column's numerical clues. For example, in a 10-cell row with a clue of "8", the middle 6 cells must be filled regardless of placement. This reasoning requires no information from other rows or columns and is relatively straightforward to implement. However, when a puzzle cannot be fully solved by line logic alone, hypothesis-and-backtracking across rows and columns becomes necessary — temporarily assuming a cell's state and verifying through cascading effects whether the assumption is globally consistent. This kind of joint reasoning requires the model to simultaneously maintain multiple constraints across the entire grid, placing higher demands on effective context window utilization and the stability of multi-step logical chains. It is also the fundamental reason why most models fail in hard mode.

Results: Accuracy Drops Sharply as Grid Size Increases

The most striking finding is a strong negative correlation between puzzle size and model performance. Looking at each model's best effort level:

• 5×5 grids: ~85% solve rate
• 10×10 grids: drops to 46%
• 15×15 grids: falls to just 20%

In other words, each step up in grid size makes it more likely for a model's logical chain to collapse midway.

Among top-performing models, GPT-6 Astra solved all 30 standard mode puzzles, delivering the most consistent performance. In the more demanding hard mode, Claude Opus 5.5 solved 8 out of 10 puzzles, making it the strongest model on 20×20 grids. In stark contrast, 11 out of 15 models in hard mode solved zero puzzles — demonstrating that problems requiring genuine cross-row-and-column joint reasoning remain a blind spot for the vast majority of models.

An Overlooked Detail: Output Format Can "Trip Up" Models

Testing also exposed an interesting engineering issue. When models were originally asked to return answers as a single 400-character string, most would "miscount cells" before the logic even got difficult — meaning they lost positional tracking while maintaining the long grid state.

For this reason, hard mode was changed to have models return an array of 20 row strings instead of a single long string. This change is revealing: model reasoning failures sometimes stem not from flawed logic itself, but from difficulty maintaining precise positional tracking within an overly long, unsegmented sequence. For any application requiring models to handle structured grids or matrices, this is a practical lesson worth heeding — output format design has a real, measurable impact on final accuracy.

This phenomenon has a parallel concept in cognitive science: chunking. When humans process long sequential information, they naturally divide it into meaningful smaller units to reduce working memory load. Large language models face analogous constraints during autoregressive generation: when tracking a precise position within a 400-character unsegmented sequence, the model is implicitly maintaining "which cell am I on" through next-token prediction probabilities, which is highly prone to drift. Structuring output as 20 independent row strings effectively provides the model with natural checkpoints — resetting the counter after each completed row, dramatically reducing the probability of cross-row positional errors. This finding has direct engineering implications: when asking models to fill tables, generate code matrices, or handle any two-dimensional structured data, prioritize segmented or nested output formats over a single flat string.

Limitations and Openness

The authors are honest about the benchmark's limitations. Since each puzzle is attempted only once, individual results carry noise — the website displays 95% confidence intervals to caution readers against over-interpreting specific numbers. While the single-attempt design increases noise, it also more closely mirrors real-world usage where a single query produces the final answer.

The project's greatest value lies in its complete openness. The testing website is publicly accessible at nonobench.com, and all code is open-sourced under the MIT license at github.com/mauricekleine/nonobench, allowing anyone to reproduce, extend, or evaluate new models.

For researchers and developers focused on evaluating model reasoning capabilities, Nonobench provides a clean, controlled testing environment that is difficult to contaminate with training data. Nonogram puzzles are less susceptible to data leakage than math problems, while cleanly quantifying a model's reasoning decay curve across different scales. It may not be the most comprehensive benchmark, but it's a sharp, information-dense complement to existing evaluations.