BrickBench: Testing AI Agents' Design Skills with LEGO Bricks

When AI Agents Meet LEGO: A Brand-New Evaluation Benchmark

How do we measure AI agents' design capabilities in the real physical world? BrickBench, from academia, offers an intriguing answer—have agents design LEGO brick sets. This work proposes an agentic benchmark for text-conditioned LEGO set design, requiring AI to select bricks from a discrete parts library based on text prompts and build assemblies that satisfy both semantic and design criteria while being physically buildable.

Unlike traditional text generation tasks, LEGO design poses unique challenges for agents: it requires not only understanding semantic instructions like "build a bridge" or "make a car," but also handling complex trade-offs between local connection constraints (how bricks interlock) and global structural constraints (whether the whole is stable and well-proportioned).

Text-Conditioned Generation refers to a model taking natural language descriptions as input conditions and generating target outputs that satisfy those descriptions, where outputs can take the form of images, code, 3D structures, and more. Compared with unconditional generation, it requires the model not only to produce reasonable results but also to remain consistent with textual semantics. In BrickBench, text conditioning manifests as prompts like "build a bridge," where the agent must map abstract semantics to concrete brick selections and spatial layouts. This process involves cross-modal understanding: the language description itself does not specify exact geometry, so the agent must infer a reasonable structural plan on its own. Evaluation of text-conditioned generation typically requires measuring semantic alignment, and the LEGO task further adds physical buildability constraints, making it more challenging than image or text generation.

How Does BrickBench Evaluate AI's Building Ability?

The research team designed three evaluation scenarios at different levels of difficulty, varying in scale and available parts libraries, and scores agents across multiple dimensions. The evaluation metrics cover three core aspects:

• Validity: Whether the generated design is physically buildable, meaning bricks can actually connect to one another with no floating or conflicting parts
• Alignment: Whether the build result faithfully reflects the semantic requirements of the text prompt
• Design quality: The aesthetics, stability, and creativity of the overall structure

This multi-dimensional scoring system avoids the one-sidedness of a single metric. An agent might produce a structure that "stands" but looks nothing like the object described in the prompt; conversely, it might match semantics well but be structurally loose and impossible to actually build.

Constraint Satisfaction Problem (CSP) refers to finding an assignment of values to a set of variables under a set of constraints such that all constraints are satisfied. LEGO building is naturally a CSP: the position and orientation of each brick are variables, while interlocking rules, non-overlap, and connectivity requirements form constraints. Unlike classical CSPs, the LEGO task also includes soft constraints (such as aesthetics and proportional harmony), expanding the problem from "does a solution exist" to "find the optimum among solutions satisfying hard constraints." Agents must make sequential decisions in this process: each placement affects the subsequent available space, and a local optimum may lead to global failure. This combinatorial explosion makes LEGO an ideal testbed for evaluating agents' planning and search capabilities.

BrickAgent: A Building Environment Designed for Coding Agents

Alongside the benchmark, the team also released BrickAgent—a development environment specifically designed for coding agents. In this environment, agents can construct, inspect, and verify their own designs. The significance of this environment lies in transforming the LEGO building task into a programmable, iterative engineering problem, where agents can manipulate bricks through code, run verification logic, and make corrections based on feedback.

This approach extends agents' capabilities from pure text reasoning to spatial reasoning and constraint satisfaction, bringing them closer to the cognitive demands of engineering design in the real world.

Coding Agent refers to an AI system capable of writing, executing, and debugging code to accomplish tasks, typically equipped with tool use, code interpretation, and iterative correction abilities. Unlike purely conversational agents, coding agents operate on environments through programmatic interfaces, enabling precise control of each action and access to structured feedback. In BrickAgent, agents use code to select bricks, specify positions, run verification logic, and adjust plans based on error messages. This "write-execute-verify-correct" loop turns the design task into an iterative engineering workflow, allowing agents to handle complex constraints beyond single-pass reasoning. Evaluation of coding agents therefore focuses not only on final outputs but also on their debugging and error-correction process.

Performance of Leading Agents: Passing but Not Excellent

The research yielded a thought-provoking finding: current leading AI agents "basically satisfy" verifiable physical and semantic requirements, but the designs they generate still show a clear gap compared with the work of human designers.

This means agents can already handle hard constraints—bricks connect, structures hold, semantics roughly match—but on soft dimensions such as design taste, structural elegance, and creative expression, they remain far from human level. This "meets the bar but fails to shine" performance precisely reveals the capability boundary of current AI in open-ended design tasks.

Why Is the LEGO Benchmark Worth Watching?

The value of BrickBench lies not only in testing AI's ability to build with bricks. It actually constructs a composite task combining discrete choices, spatial reasoning, and physical constraints—tasks that are widespread in real-world robot manipulation, automated design, and engineering planning.

The discreteness of LEGO parts and the strictness of connection rules make it an ideal test platform: agents must make sequential decisions within a limited action space while balancing local connection correctness and global structural soundness. If this capability can be effectively improved, it will transfer directly to broader embodied intelligence and automated design scenarios.

The research team has open-sourced the benchmark and development environment at http://www.brickben.ch, providing reproducible infrastructure for follow-up research. For researchers focused on agent evaluation, spatial reasoning, and AI design capabilities, this is a new direction worth exploring in depth.

One-line Summary

BrickBench uses LEGO building tasks to evaluate AI agents' physical and semantic design capabilities, finding that they can meet the bar but fail to shine.

Paragraph Summary

BrickBench is an agentic benchmark for text-conditioned LEGO set design, requiring AI to select bricks from a discrete parts library based on text prompts and construct assemblies that are both semantically aligned and physically buildable. Evaluation covers three dimensions—validity, alignment, and design quality—across three difficulty scenarios. The accompanying BrickAgent environment allows coding agents to build, inspect, and verify designs, turning the task into a programmable engineering problem. The study finds that current leading agents basically satisfy hard physical and semantic constraints but still clearly lag behind humans in design taste, structural elegance, and creative expression. Combining discrete choices, spatial reasoning, and physical constraints, the benchmark has transfer value for embodied intelligence and automated design research and has been open-sourced.