CAD World Benchmark: Why AI Agents Succeed Only 17.5% of the Time at Engineering Drawing

The Long-Missing CAD Automation Benchmark Has Arrived

Computer-use agents have been advancing rapidly. These systems can interpret software interfaces and autonomously control the mouse and keyboard to execute user instructions. Unlike traditional tools that call specific APIs, their core approach simulates how humans interact with graphical interfaces. But most existing benchmarks have focused on web browsing and general desktop applications, with little coverage of specialized engineering software like CAD.

A new study called CAD World specifically evaluates long-horizon tasks in the mechanical CAD domain, making it the first computer-use benchmark dedicated to mechanical CAD workflows. Automation in this space has always been challenging — CAD demands far greater operational precision and state tracking than ordinary software, making it an ideal testbed for revealing the true capability limits of current agents.

The results are striking: across seven mainstream computer-use agents running all 200 tasks, the best-performing model — GPT-5.4 — achieved only a 17.5% success rate, compared to 87% for human experts. That enormous gap is exactly what CAD World was designed to expose.

The technical approach of computer-use agents is fundamentally different from traditional RPA (Robotic Process Automation). RPA relies on pre-recorded scripts or hardcoded UI element selectors — any software update or layout change breaks it. Computer-use agents instead use vision-language models (VLMs) to interpret screenshots in real time and dynamically decide on the next action, essentially "looking at the screen and thinking" rather than following a fixed script. This theoretically lets them handle interface changes and novel scenarios, but introduces new challenges: long reasoning chains, error accumulation, and difficulty maintaining operational precision. This is precisely why high-precision professional software like CAD is so effective at exposing their limitations.

Why CAD Is Exceptionally Hard

Drawing a single mechanical part may require hundreds of steps, and one mistake can cascade through everything that follows. The paper identifies several core challenges unique to the CAD domain.

Reasoning about 3D structure from a 2D interface is the first hurdle. Two lines that look adjacent on screen may or may not be coincident or parallel — understanding these geometric relationships is far from trivial. Extreme operational precision is the second challenge: clicking the wrong button or entering a wrong parameter corrupts the entire constraint system, and errors propagate downstream in ways that are very difficult to undo.

The remaining two challenges are equally critical. Real CAD workflows are long-horizon by nature, often spanning dozens to hundreds of steps across sketching, feature trees, assembly, simulation, and other modules. Agents must remember prior operational state and adapt their strategy accordingly. And critically, correctness cannot be judged by appearance alone: two models that look identical might differ by 0.1 mm in dimension or have different constraint relationships — differences that cause failures in downstream manufacturing or simulation. A valid model must be editable, measurable, and compliant with engineering specifications, a far higher bar than simply completing surface-level interactions.

Many prior AI studies in the CAD space took the route of generating CAD code or geometric models directly, bypassing the GUI entirely. But in practice, engineers work through the GUI — they don't write code to generate models. Skipping to code generation omits the most essential part of the interaction: how to locate the feature that needs to be modified in the interface, and how to verify the result afterward. CAD World's core insight is to make agents interact with the same GUI that human engineers use, testing whether they can perceive, operate, and verify engineering state through the interface.

Parametric modeling is the foundational paradigm of modern mechanical CAD software, fundamentally different from early direct modeling approaches. In parametric modeling, a model is defined by an ordered sequence of feature operations stored in a Feature Tree, each preserving modifiable parameters and geometric constraints. Changing a parameter in an early step causes all downstream features that depend on it to automatically recompute. This makes engineering models editable by design — but it also means operation order is critical. Executing the right command in the wrong state, or entering the wrong parameter at the right step, causes the constraint solver to fail or produce invalid geometry. Agents must not only know "what to do" but also understand the current state of the feature tree and recognize which modeling stage they are in — placing extraordinary demands on contextual state tracking.

200 Tasks and Four Evaluation Properties

CAD World comprises 200 tasks spanning 11 workflow categories and 183 distinct knowledge points, covering the full mechanical design pipeline: sketching, part modeling, assembly, manufacturing (CAM), programming, simulation, measurement, engineering drawings, and more. Sketching accounts for 63 tasks (60 knowledge points) and part modeling for 77 tasks (61 knowledge points), with the remainder covering practical engineering operations like chamfers and fillets that many prior CAD datasets rarely include.

Every task is fully configured with a natural language instruction, an initial software state, optional reference materials, and an automatically executed evaluator. Agents can only perceive the interface through screenshots and output mouse and keyboard actions. Final evaluation is based on whether the saved CAD file itself is correct — not on textual answers or visual similarity. Even if a model looks like a triangular prism on screen, it still counts as a failure if its structure, geometric parameters, or modeling process don't meet requirements.

The evaluators are designed around four properties: traceable (all screenshots, logs, and evaluation results can be correlated to pinpoint exactly where things went wrong); measurable (dimensions, constraints, and simulation results are checked automatically); diagnosable (failures can be attributed to missing objects, attribute mismatches, tolerance violations, or execution errors); and editable (saved files can be opened and modified directly in the original software). To achieve this, the authors chose the open-source FreeCAD as the runtime environment — freely distributable, with full GUI support, and easy for other researchers to reproduce.

The entire annotation and validation process took roughly three months and 1,300 person-hours. The median instruction length is 63 words, with tasks sourced from FreeCAD workbench functions, CAD tutorial documentation, and common mechanical design workflows.

Behind 17.5%: Failure Shifts Later in the Pipeline

All model runs were conducted in a unified CAD World runner, with FreeCAD running inside a pre-built Ubuntu virtual machine managed by Docker and QEMU. Agents have no special privileges — no direct file access, no API calls — and each task allows a maximum of 100 steps.

The performance gap is stark. Human experts achieved an 87% success rate using an average of just 26.5 steps. The best model, GPT-5.4, reached only 17.5% success, with a 62.5% completion rate and an average of 104.6 action steps — and even successful tasks averaged 46.9 steps, nearly twice what humans needed. The second-ranked Opus-4.8 achieved 16% success at an average cost of $10 per task. Other models fared worse: Kimi K2.6 reached 7.5%, OpenCUA just 1.5%, and two models scored exactly 0% — with these low-performing models actually taking more steps, largely spinning in ineffective actions or getting stuck in loops.

One instructive pattern emerged: as model capability increases, failure shifts to later stages rather than disappearing. Low-performing models mostly fail at execution — unable to produce valid artifacts at all. Stronger models like GPT-5.4 and Opus-4.8 can generally save readable CAD projects, but their failures stem from document structure errors, geometry errors, and incorrect modeling workflows. This suggests that once general UI interaction is solved, the next bottleneck is maintaining engineering semantic consistency across the full workflow — not just knowing how to click, but understanding the logic and rules of engineering design.

How CAD World Differs from Existing Benchmarks

General computer-use benchmarks like WebArena focus primarily on web interaction; OSWorld extends to ordinary desktop applications; OSWorld 2.0 begins to address longer workflows — but CAD remains a small fraction with no systematic coverage of sketch constraints, parametric modeling, assembly, CAM, or other specialized content. Dedicated CAD benchmarks like ABC, Fusion360Gallery, and Text2CAD mostly test geometry or code generation, requiring no GUI interaction and covering only short workflows.

CAD World simultaneously satisfies all of the following dimensions: GUI interaction, native editability, precise parametric representation, expert-authored tasks, task-specific evaluation, and long-horizon workflows (averaging over 40 steps each) — filling a genuine gap in the field. Its value extends beyond benchmarking: every task includes an expert demonstration trajectory, full state information, and evaluation criteria, which can be used to fine-tune specialized CAD operation agents.

The researchers are candid about how large the gap remains. A 17.5% success rate is barely a starting point, far from commercial viability. Real industrial design scenarios can involve thousands of steps, spanning cross-module collaboration and industry standards. But the methodology of CAD World can be applied to other professional software domains — electronic design automation, architectural design, audio/video editing — helping computer-use agents move from general-purpose to domain-specific settings. What's most commendable about this work is that the authors didn't simplify tasks or grant models special privileges to inflate success rates. They exposed the full difficulty of real industrial scenarios to the AI research community — because only by confronting genuine challenges can technology actually reach deployment.

FreeCAD is an open-source parametric CAD package built on the OpenCASCADE geometric kernel, supporting multiple workbenches for part design, assembly, FEM simulation, and CAM path planning. It is one of the most fully featured open-source mechanical CAD platforms available. Compared to commercial software like SOLIDWORKS or Fusion 360, FreeCAD is freely distributable, supports Python scripting for automation, and exposes internal model state through command-line or API access — providing the technical foundation for CAD World's automated evaluators to perform precise geometric checks (dimensions, constraint status, topology). Choosing FreeCAD also means the benchmark can be reproduced locally by any researcher without a commercial software license, significantly lowering the barrier for future community research.