A Real-World Engineering CAD Segmentation Benchmark: Rule-Based vs. Learning-Based Methods for Hole Detection

Why CAD Segmentation Needs Real-World Benchmarks

CAD (Computer-Aided Design) segmentation is a critical component of modern engineering software — it determines whether machines can automatically understand the geometric structure of a 3D model. Yet most mainstream CAD segmentation datasets are either synthetically generated or automatically annotated, which raises a long-overlooked question: how do these methods actually perform on real engineering tasks?

A new study addresses this gap head-on. The researchers focus on fastener-relevant hole detection as their entry point — a representative real-world engineering task that requires manual annotation and involves far greater geometric complexity than typical test benchmarks. In other words, it more closely resembles the problems engineers face in practice, rather than idealized laboratory data.

A Head-to-Head Comparison of Two Approaches

The core of the study is a direct comparison between two major technical paradigms: rule-based methods and learning-based methods.

Rule-Based Methods: Robust Without Training Data

The research team developed two rule-based approaches for hole detection: graph-wire-search and multi-view. Both represent the traditional geometric reasoning paradigm — using pre-designed logical rules to directly identify target features from a model's Boundary Representation (B-Rep).

Experiments revealed distinct behavioral tendencies in each approach: the graph-wire-search method tends toward higher recall, aiming to miss as few holes as possible; the multi-view method tends toward higher precision, producing more reliable detections with fewer false positives. This recall-versus-precision tradeoff has direct practical implications — different production environments have very different tolerances for "false alarms" versus "missed detections."

B-Rep (Boundary Representation) is the most prevalent 3D geometric representation in CAD, describing solid models through oriented faces, edges, vertices, and their topological relationships. Unlike voxels or point clouds, B-Rep preserves precise geometric semantics (e.g., planar, cylindrical, and toroidal surfaces), making it well-suited for rule-based feature recognition. The graph-wire-search method leverages B-Rep's topological graph structure: holes typically correspond to loops of connected arc edges, and candidate holes can be located by searching for subgraphs that satisfy specific constraints within the topology graph. The multi-view method takes a different approach — it projects the 3D B-Rep into 2D contours from multiple orthogonal views, detects circular features in 2D, then back-projects the results into 3D space, sidestepping the complexity of directly processing 3D topology.

Learning-Based Methods: High Performance but Sensitive to Domain Shift

The second paradigm is deep learning. The study introduces two representative B-Rep segmentation models — UV-Net and AAGNet — and trains them on the newly constructed, manually annotated benchmark dataset.

Results show that learning-based methods achieve strong performance when sufficient annotated data is available. However, they also expose a clear weakness: robustness degrades significantly under domain shift. That is, when the distribution of test data differs from the training data, model performance drops — a situation that arises frequently in real engineering environments, where CAD models vary widely across companies and product lines.

UV-Net is a graph neural network designed specifically for B-Rep. It parameterizes each face as a feature map in UV coordinate space and aggregates global information through a face adjacency graph, enabling face-level semantic segmentation. AAGNet (Attributed Adjacency Graph Network) builds on this by enriching edge feature representations — encoding geometric relationships between faces (e.g., convex/concave transitions, shared edge types) as graph edge attributes, allowing the model to capture finer-grained assembly and feature semantics. Both models rely on large amounts of annotated B-Rep data to learn feature representations, which is precisely the root of the domain shift problem: CAD models from different sources vary significantly in modeling conventions, feature granularity, and parameterization style, making representations learned on one dataset difficult to transfer directly to another.

Rigorous Experimental Design

The value of this study lies not only in its conclusions but also in its systematic experimental design. The team conducted evaluations across four dimensions:

• Parameter optimization: Finding optimal configurations for rule-based methods;
• Benchmark dataset evaluation: Testing on manually annotated real-world engineering data;
• Cross-domain generalization: Validating transferability on synthetic datasets;
• Feasibility assessment: Examining the applicable boundaries of each method under varying dataset sizes and IoU (Intersection over Union) thresholds.

This multi-angle cross-validation ensures that findings are not artifacts of a single scenario, but carry broader reference value. The sensitivity analysis with respect to data scale is particularly valuable — it directly answers the question engineering teams care about most: how much annotated data is actually needed before a learning-based approach becomes worthwhile?

IoU (Intersection over Union) is the core metric in segmentation tasks for measuring the overlap between predicted results and ground-truth annotations. It equals the area of intersection divided by the area of union between the predicted and annotated regions, ranging from 0 to 1 — higher values indicate more accurate predictions. In CAD segmentation, the choice of IoU threshold is especially critical: hole boundaries in engineering parts often contain minor geometric errors or modeling ambiguities. Thresholds set too high may cause practically usable predictions to be incorrectly marked as failures, while thresholds set too low may mask genuine shortcomings. Systematically evaluating method performance across multiple IoU thresholds more accurately reflects real-world utility within engineering tolerances, and helps different application scenarios make targeted choices based on their precision requirements.

Core Insight: The Tradeoff Between Data Requirements and Robustness

The study's key findings are clear and pragmatic. Rule-based methods generalize well without any training data, making them suitable for data-scarce scenarios or situations requiring rapid deployment. Learning-based methods deliver stronger performance when data is abundant, but pay a cost in robustness under domain shift.

This reveals a fundamental tension that runs throughout the deployment of machine learning in practice — the tradeoff between data requirements and robustness. Neither approach is an absolute winner; the right choice depends on the constraints of the specific task: How much annotation budget do you have? Is your data distribution stable? Are you more concerned about missed detections or false positives?

More importantly, the researchers emphasize that these findings extend beyond hole detection and can guide method selection for a broader range of real-world CAD segmentation tasks. For teams working to integrate AI into engineering design workflows, this is a rare, evidence-based decision-making reference.

Conclusion

In a research ecosystem dominated by synthetic data, this work brings the focus back to the complexity of real engineering practice. It neither blindly champions deep learning's superiority nor rigidly defends the reliability of traditional rule-based methods — instead, it uses rigorous comparative experiments to map the boundaries and complementarity of both. For the rapidly advancing field of intelligent CAD, understanding which method works under which conditions is far more meaningful than chasing a single performance number.