LLM-Generated CAD Assemblies Look Good but Don't Work? FunCAD Framework Closes the Loop on Functional Validation with Physics Simulation

Visually Plausible ≠ Functionally Usable: The Core Dilemma of CAD Agents

LLM-driven agents can already generate CAD assemblies that look quite professional, but according to this research, these designs often fail to achieve their intended physical function. The problem isn't that the model can't draw a plausible structure — it's that there's a gap between "looks right" and "actually works."

The research team defines this scenario as functional CAD design: the agent must understand a request with incomplete information, build a mechanism that meets the functional requirements of a physics simulation, and iteratively refine it. Unlike conventional CAD generation tasks, the acceptance criterion here isn't the aesthetic quality of a rendered image, but whether the mechanism can complete specified motions in simulation.

The tricky part of this problem is validation itself: the same function can be implemented in countless ways — different part combinations, different control strategies, different motion sequences can all achieve the same effect. Checking against fixed templates easily leads to misjudgment.

CAD assemblies refer to 3D models composed of multiple parts combined under certain constraint relationships. In traditional CAD workflows, an assembly not only describes geometry but also implicitly encodes mating relationships (e.g., shaft-hole fits, gear meshing) and degrees of freedom. When LLM agents generate assemblies, they typically output part geometry and assembly constraints as text or code, which a CAD kernel then instantiates. But geometric validity does not equal kinematic solvability — for example, parts may have undeclared interferences, or insufficient constraints may leave the mechanism under-defined. Understanding this is key to seeing why "professional-looking render" and "mechanism that can actually move" are two independent evaluation systems.

Physics simulation in the CAD context usually refers to rigid-body dynamics simulation: applying drives to a mechanism (e.g., motor torque or displacement) and numerically integrating to solve for each part's motion trajectory, contact forces, and joint reaction forces. Simulation can detect motion interference, singular configurations, timing conflicts, and other issues that visual inspection cannot catch. Unlike rendering, simulation requires precise mass properties, friction coefficients, and constraint definitions — any omission leads to distorted results. This is why connecting LLM output to a simulator often requires filling in physical parameters, and one reason why "simulation-augmented" baselines show limited effectiveness.

FunCAD-Harness: Translating Abstract Requirements into "Candidate-Specific" Physical Tests

To address the validation bottleneck, the researchers propose the FunCAD-Harness framework. Its core idea: rather than presupposing a unified test standard, generate a physics test tailored to each candidate design, then use simulation feedback to guide the next round of revision.

This design sidesteps the mindset of "one function, one standard answer." Since different mechanisms may implement the same function in completely different ways, tests must follow the candidate — only a customized validation scheme can expose real problems.

According to the research description, this approach can reveal subtle mechanical failures that visual inspection completely misses. This means a mechanism that appears to run smoothly in a screenshot may fail in simulation due to motion interference, timing misalignment, or force-related issues — problems that only physics simulation can expose.

Candidate-specific testing draws inspiration from property-based testing in software engineering: rather than specifying a single correct output, generate validation conditions for each candidate that can distinguish correct from incorrect. In mechanism design, this means automatically deriving the motion sequence the mechanism should complete and its pass criteria based on the candidate's topology and actuation, then handing them to simulation for execution. The difficulty is that the test generator itself must understand mechanism semantics; otherwise it may produce tests that are too loose (any design passes) or too strict (even correct designs fail) — which is exactly why this framework requires dedicated design effort.

FunCADBench: Systematic Evaluation Across 46 Mechanism Families

To systematically evaluate this capability, the team built the FunCADBench benchmark, covering 46 mechanism families, with these mechanisms generally involving interdependent multi-step operations.

Multi-step, tightly coupled operation is a key feature of this benchmark: failure at one stage cascades into subsequent motions, causing "roughly right" designs to quickly reveal themselves during testing. Experiments on frontier LLM agents show that compared to standard simulation-augmented baselines, FunCAD-Harness significantly improves functional success rates.

This also indirectly shows that simply attaching a simulator behind an LLM isn't enough — what matters is how tests are generated and how feedback is utilized.

Mechanism families refer to mechanisms that belong to the same class in terms of topology or motion principle, such as four-bar linkages, gear trains, cam mechanisms, and intermittent motion mechanisms. Organizing benchmarks by family allows testing whether an agent has truly mastered the kinematics of a class of mechanisms rather than merely memorizing individual cases. The 46 families span the spectrum from simple linkages to multi-stage transmissions, and dimensions and layouts can vary within a family, increasing generalization difficulty. This layered evaluation reveals model capability boundaries better than a single task set.

From "Generating Drawings" to "Delivering Usable Mechanisms": How Far to Go?

The significance of this work is advancing the evaluation criteria for CAD agents from the visual level to the functional level. When agents can autonomously revise designs based on simulation feedback, we are one step closer to "translating human intent into mechanisms that actually run."

Of course, 46 mechanism families are just a starting point — constraints in real engineering scenarios are far more complex. But the approach FunCAD-Harness represents — replacing fixed acceptance templates with candidate-specific physical tests — provides a reusable methodological framework for future research. For those interested in deploying AI in engineering design, functional validation rather than visual generation may be the next direction worth investing in.

One-line Summary

LLM-generated CAD assemblies look plausible but struggle to pass physical functional validation; FunCAD-Harness improves functional success rates with candidate-specific simulation tests.

Paragraph Summary

The article notes that LLM-driven CAD agents can generate professional-looking assemblies but often fail physical functional validation, with the core dilemma being the gap between "looks right" and "actually works." To address this, researchers propose the FunCAD-Harness framework, which instead of presupposing a unified test standard, customizes physical tests for each candidate design and uses simulation feedback to guide iterative revision, thereby exposing subtle failures like motion interference and timing misalignment that visual inspection cannot detect. The team also built the FunCADBench benchmark covering 46 mechanism families involving interdependent multi-step operations. Experiments show that compared to standard simulation-augmented baselines, FunCAD-Harness significantly improves functional success rates, indicating that how tests are generated and how feedback is utilized are the key factors.