DishwasherBench: Can Frontier Models Load a Dishwasher Properly?

The landscape of AI benchmarks has expanded once again. A new test called DishwasherBench has appeared on Hacker News, with a name so blunt it borders on absurd: let frontier models judge how a dishwasher should actually be loaded.

Why a dishwasher?

Open any model leaderboard and you'll see math competition problems, code generation, legal exams, medical Q&A, and PhD-level physics questions. What these tests share is that they compress intelligence into text input and text output — the model only needs to "answer," not "act."

DishwasherBench takes a different path. Loading a dishwasher may seem trivial, but it requires genuine spatial reasoning about the real world: which rack the pots go on, which way bowls should face, the tilt angle of cups, how to partition the cutlery basket, and how large and small items occlude one another. It's an optimization problem under three-dimensional constraints, and the model can only decompose it using visual input and common sense.

Making a household chore into a benchmark isn't a joke at its core. Real-world scenarios often expose a model's capability boundaries more effectively than exam questions — whether it truly understands "relationships between objects," rather than reciting answers it has seen in training data.

Spatial reasoning is a core concept in cognitive science and AI research, referring to an agent's ability to represent and infer the position, orientation, size, occlusion relationships, and motion trajectories of objects in a three-dimensional environment. Humans rely on intuitive physics when handling such tasks — an implicit knowledge that predicts how objects interact without requiring formulas. For models, spatial reasoning is far harder than textual reasoning, because language can represent knowledge serially, whereas spatial relationships are parallel and continuous. Although current mainstream multimodal models can recognize objects in images, they still frequently fail at tasks requiring tracking mutual constraints among multiple objects and anticipating the consequences of motion. This is also one of the long-standing bottlenecks in robotics and embodied AI.

The real state of frontier models on spatial reasoning

What's most noteworthy about this test isn't the scores themselves, but the question it exposes that the model community has long avoided: the gap between language ability and physical common sense.

A model can accurately explain the second law of thermodynamics in a paper abstract, yet may not be able to judge whether a deep pot placed on the upper rack of a dishwasher will block the rotating spray arm. The former is text pattern matching; the latter requires constructing a rotating three-dimensional space in one's head.

What makes DishwasherBench valuable is precisely that it anchors evaluation in a task the model cannot solve by "memorization." With the same set of objects, the same set of rules, and the same set of constraints, the model must make a series of mutually influencing decisions. Any local error propagates to the whole — this is highly isomorphic to planning problems in the real world.

A multimodal model is one capable of processing multiple input forms such as text, images, and audio simultaneously. Compared with pure language models, they theoretically possess the ability to understand visual scenes, and are therefore expected to handle spatial tasks. However, research shows that multimodal models exhibit a clear "language shortcut" phenomenon in visual understanding: when image information conflicts with textual priors, models often preferentially rely on language patterns in training data rather than what they actually see. This means that even if a model can accurately describe a photo of a dishwasher, it may not be able to mentally simulate the conflict between the spray arm's rotation trajectory and the placement of pots. DishwasherBench precisely exposes this capability gap of "seeing but not inferring."

Benchmarks are moving from "taking exams" to "doing things"

If you trace the evaluation trends of the past two years into a single line, you'll see a directional shift. Early benchmarks focused on knowledge Q&A, followed by code racing, Agent tasks, web operation, and now household chore simulation. The evaluation target is gradually shifting from "what the model knows" to "what the model can accomplish."

The driving force behind this shift comes from the application side. As AI begins to be integrated into robots, smart homes, and automated workflows, scores on pure text can no longer answer the core question: can it do one small thing correctly in the physical world?

DishwasherBench is an early sample of this wave of "embodied" evaluation. Its questions look light, but the problem it points to is heavy — if evaluation standards remain on paper, model capabilities will remain on paper.

Embodied evaluation is an emerging direction in the AI evaluation field in recent years. The core idea is to have models complete tasks in simulated or real physical environments, rather than merely answering in textual space. The classic definition of embodied intelligence emphasizes that an agent needs to acquire knowledge through interaction between body and environment, and this concept is now being introduced into model evaluation. Unlike traditional static Q&A benchmarks, embodied evaluation typically involves action sequence planning, environmental state changes, and causal inference, and task correctness depends on the final environmental state rather than intermediate answers. The rise of this type of evaluation is directly related to AI's penetration into fields such as robotics, smart homes, and autonomous driving, because in these scenarios, the gap between "knowing" and "doing" is often the key to product success or failure.

Where will such benchmarks go?

Household scenarios are naturally suited for evaluation because the rules are clear, results are verifiable, and variables are controllable. After the dishwasher, we'll likely see more similar designs: organizing the refrigerator, folding and stacking clothes, packing and moving boxes. They share one characteristic — humans can judge right from wrong in seconds, but machines require complete spatial and logistical reasoning.

The real test is whether the scoring system can remain stable. There is no single correct answer for loading a dishwasher — as long as everything fits, gets clean, and doesn't block the spray arm, multiple approaches work. If the benchmark can only determine whether a solution "matches the reference solution," then what it measures isn't capability but alignment. This may be the question DishwasherBench needs to answer next.

For developers and researchers concerned with the true boundaries of model capability, these kinds of "unseemly" tests are actually more worth tracking than yet another academic leaderboard.

One-line Summary

DishwasherBench uses dishwasher-loading tasks to reveal models' spatial reasoning weaknesses, signaling AI evaluation's shift from exam-taking to doing.

Paragraph Summary

DishwasherBench is a new AI benchmark that appeared on Hacker News, requiring frontier models to judge how pots and dishes should be arranged in a dishwasher. This seemingly absurd design actually points to a serious issue: a clear gap exists between models' language ability and physical common sense. Dishwasher loading involves three-dimensional spatial constraints, occlusion relationships, and motion trajectory prediction, which models cannot solve by reciting training data. The article notes that this reflects AI evaluation's shift from knowledge Q&A toward task completion, and household scenarios — with clear rules and verifiable results — may become a common template for embodied evaluation. But such benchmarks also face the challenge of unifying scoring criteria — loading solutions are often not unique, and if only compared against a reference solution, what's measured may be alignment rather than true capability.