Nonobench
An open-source benchmark project for large language models that evaluates structured logical reasoning and spatial memory by requiring models to solve nonogram (picross) puzzles. The benchmark covers grid sizes ranging from 5×5 to 20×20, spanning multiple difficulty levels.
Timeline (last 90 days)
Nonobench is an open-source benchmarking project that has 49 large language models solve nonogram/picross puzzles.
The Nonobench testing rules require each model to receive row and column clues only once, return the complete grid in a single response, without calling external tools, with only one attempt per puzzle.
Nonobench standard mode contains 30 puzzles with grid sizes increasing from 5×5 to 15×15, with puzzles taken from the Nonograms dataset published by Moyà-Alcover (licensed under CC BY 4.0).
Nonobench hard mode contains ten randomly generated 20×20 large grids, each validated to ensure a unique solution, with five of them unsolvable by line logic alone.
Hard mode grids use random fill rather than real patterns, with the purpose of preventing models from guessing the answer based on image priors.
The test ran a total of 130 variants, covering different levels of reasoning effort, called via OpenRouter with official endpoints for each lab locked in wherever possible.
Taking each model's best effort tier, the solve rate for 5x5 grids is approximately 85%, dropping to 46% for 10x10, and only 20% remaining for 15x15.
Puzzle size shows a strong negative correlation with model performance, and scaling up the grid causes a sharp cliff-like drop in accuracy.
GPT-6 Astra solved all 30 puzzles in standard mode, demonstrating the most robust performance.
Claude Opus 5.5 solved 8 out of 10 puzzles in hard mode, making it the strongest performer on 20x20 large grids.
4 more timeline events
All Facts (14)
Nonobench is an open-source benchmarking project that has 49 large language models solve nonogram/picross puzzles.
50%UnverifiedThe Nonobench testing rules require each model to receive row and column clues only once, return the complete grid in a single response, without calling external tools, with only one attempt per puzzle.
50%UnverifiedNonobench standard mode contains 30 puzzles with grid sizes increasing from 5×5 to 15×15, with puzzles taken from the Nonograms dataset published by Moyà-Alcover (licensed under CC BY 4.0).
50%UnverifiedNonobench hard mode contains ten randomly generated 20×20 large grids, each validated to ensure a unique solution, with five of them unsolvable by line logic alone.
50%UnverifiedHard mode grids use random fill rather than real patterns, with the purpose of preventing models from guessing the answer based on image priors.
50%UnverifiedThe test ran a total of 130 variants, covering different levels of reasoning effort, called via OpenRouter with official endpoints for each lab locked in wherever possible.
50%UnverifiedTaking each model's best effort tier, the solve rate for 5x5 grids is approximately 85%, dropping to 46% for 10x10, and only 20% remaining for 15x15.
50%UnverifiedPuzzle size shows a strong negative correlation with model performance, and scaling up the grid causes a sharp cliff-like drop in accuracy.
50%UnverifiedGPT-6 Astra solved all 30 puzzles in standard mode, demonstrating the most robust performance.
50%UnverifiedClaude Opus 5.5 solved 8 out of 10 puzzles in hard mode, making it the strongest performer on 20x20 large grids.
50%Unverified11 out of the 15 models that participated in hard mode failed to solve a single puzzle.
50%UnverifiedWhen models are asked to return the answer as a single 400-character string, most models miscount cells before the logic even becomes difficult, losing their ability to track positional counts.
50%UnverifiedHard mode was changed to have models return an array of 20 row strings instead of a single long string, in order to improve position tracking.
50%UnverifiedAll Nonobench code is open-sourced under the MIT license (github.com/mauricekleine/nonobench), and the test website is publicly accessible (nonobench.com).
50%