Beyond CLIP: Calibrated Decision Models for Few-Shot Defect Detection

Introduction: What Do You Do for Visual Inspection When There Are Only Two Defect Samples?

In product and process inspection within manufacturing, vision sensors have long been standard equipment. However, traditional deep learning models rely on large amounts of annotated data—often requiring thousands of images of normal and defective products for training. In recent years, vision Transformer (ViT) models represented by CLIP have significantly lowered this barrier, with research showing that the number of images required per class can be reduced to 50–100. Yet in high-yield or small-batch production scenarios, even collecting 50 defect images can be extremely difficult.

A recent paper turns its attention to a more extreme setting: inference with only k=2 reference images per class. The researchers evaluated the practicality of two structured decision models (Jev and Clef-Flash) for image classification/detection and proposed three core contributions: a k-shot learning framework, an agentic method for automatically optimizing textual descriptions, and a defect probability calibration technique.

From CLIP to Structured Decision Models: Why Is a New Paradigm Needed?

CLIP's zero-shot and few-shot capabilities stem from large-scale image-text contrastive pretraining; it guides image classification through textual descriptions. But in industrial inspection, defect definitions are often subtle and context-dependent—the boundaries between scratches, color deviations, and deformation are not always clear. Although CLIP's "System 2"-style reasoning (analogous to slow human thinking) is flexible, it may not be sufficiently stable in the few-shot regime.

The Jev and Clef-Flash models studied in the paper belong to "System One Decision Models," which emphasize fast, structured judgment paths. They likewise use textual descriptions as the basis for classification, but improve robustness with very few samples through structured decision mechanisms. This offers a possibility for high-yield production lines: deploy inspection with only two reference images, without accumulating a large number of defect samples.

System 1 and System 2 are dual-system theory concepts proposed by Daniel Kahneman in Thinking, Fast and Slow. System 1 corresponds to fast, intuitive, automatic thinking modes that consume almost no cognitive resources; System 2 is a slow, analytical reasoning process that requires volitional control. In artificial intelligence, this framework is often borrowed to distinguish two types of models: large language models and CLIP are often regarded as "System 2"-style, relying on large-scale pretraining and contextual reasoning; whereas structured decision models (such as Jev and Clef-Flash) imitate System 1's fast judgment paths and maintain stability with few samples through rule-based or template-based decision mechanisms. This analogy helps explain why, in extreme few-shot industrial inspection, structured models may be more reliable than flexible contrastive learning models—the former trade some generalization ability for determinism.

Three Core Contributions: Framework, Agent, and Calibration

k-shot Learning Framework

The researchers proposed a k-shot learning framework for image classification/detection that systematically defines how to configure Jev and Clef-Flash with very few reference images. The framework covers the organization of reference images, the construction of text prompts, and standardization of the inference pipeline, making few-shot deployment procedural.

Agentic Automatic Optimization of Textual Descriptions

Textual descriptions play a key role in decision models—they tell the model "what a defect is." But manually writing descriptions is time-consuming and experience-dependent. The paper proposes an agentic method that automatically and iteratively optimizes the textual descriptions that guide classification. Experiments show that in 6 of 8 model-application combinations, accuracy improved significantly as a result.

Defect Probability Calibration

Inspection decisions require not only classification results but also reliable probability estimates. The researchers introduced calibration methods to make the output defect probabilities better match the true likelihood. In 13 of 15 model-application configurations, the Brier score (a metric for measuring the accuracy of probabilistic predictions) was reduced, meaning the calibrated probabilities are more trustworthy and help set reasonable alarm thresholds.

The Brier score is a classic metric for evaluating the accuracy of probabilistic predictions, defined as the mean squared error between predicted probabilities and true outcomes (0 or 1). For binary classification problems, a lower Brier score indicates that the model's output probabilities are closer to the true likelihood. It penalizes both overconfidence and underconfidence: for example, if the model predicts a probability of 0.9 for a positive sample and the true label is 1, it contributes an error of 0.01; if it predicts 0.6, the error is 0.16. Unlike accuracy, which focuses only on classification thresholds, the Brier score directly measures the reliability of probabilities, so it is often used in industrial inspection scenarios that require setting alarm thresholds. Calibration methods (such as Platt scaling and isotonic regression) adjust model outputs via post-processing to align predicted probabilities with empirical frequencies, thereby reducing the Brier score.

Empirical Results from Five Manufacturing Case Studies

The paper validated the above methods in five manufacturing case studies. Under the condition of using only two reference images per class, the test accuracy ranges of Jev and Clef-Flash were 0.75–1.00 and 0.71–0.99, respectively, both outperforming CLIP's 0.67–0.84 under the same references. This result shows that structured decision models have a clear advantage in extreme few-shot scenarios.

It should be emphasized that the wide accuracy ranges reflect differences in difficulty across manufacturing scenarios. But the overall trend is consistent: Jev leads in most cases, with Clef-Flash close behind, and both consistently outperform the CLIP baseline.

In k-shot learning, k refers to the number of labeled samples available per class. In traditional supervised learning, k is usually in the tens to hundreds; k=2 is an extreme few-shot setting in which the model must induce the essential features of a class from very few examples. Compared with zero-shot learning (k=0, relying only on semantic descriptions) and one-shot learning (k=1), k=2 provides the weakest visual evidence but also introduces the risk of overfitting. In industrial inspection, the k-shot setting directly corresponds to the real-world constraint that only very few defect samples can be obtained when switching small-batch or high-yield production lines. When evaluating k-shot methods, it is usually necessary to report the average performance and variance over multiple random draws of reference images to assess stability.

Reproducibility and Tooling Ecosystem

To support reproducibility, the research team released reproduction code and published an R package called "decisions" for evaluating additional models. In addition, they provide a public web application that lets practitioners run inspection experiments with their own images without writing code. This openness lowers the barrier for industry to try new methods and helps advance standardized evaluation of few-shot defect detection.

Practical Significance and Limitations

For high-yield, small-batch, or customized production scenarios, the setting of two reference images per class is highly attractive—it means no need to stop the line to collect large numbers of defect samples when switching production. Agentic optimization of textual descriptions and probability calibration further improve deployment reliability.

However, the paper's case studies are still limited to five manufacturing scenarios, and generalization to more complex defect types (such as micro-cracks and defects in transparent materials) still needs verification. In addition, the black-box nature of System One decision models may pose challenges for root-cause analysis. Future work could explore more model combinations and cross-domain transfer capabilities.

Overall, this study provides a pragmatic path beyond CLIP for "few-shot industrial visual inspection," and its open-source toolchain is worth the attention of practitioners.

One-line Summary

Jev and Clef-Flash outperform CLIP with only 2 reference images per class, enabling few-shot industrial visual inspection.

Paragraph Summary

The paper addresses the extreme scarcity of defect samples in high-yield or small-batch production by proposing a few-shot framework for visual inspection using only k=2 reference images per class. The study evaluates two structured decision models (Jev and Clef-Flash) and contributes three techniques: a k-shot learning framework, agentic automatic optimization of textual descriptions, and defect probability calibration. In five manufacturing case studies, the test accuracy of Jev and Clef-Flash was 0.75–1.00 and 0.71–0.99, respectively, both outperforming CLIP's 0.67–0.84. Agentic optimization improved accuracy in 6 of 8 combinations, and probability calibration reduced the Brier score in 13 of 15 configurations. The study also open-sources code, an R package, and a no-code web application, providing a pragmatic path beyond CLIP for few-shot industrial inspection.