Hide and Seek: A New Black-Box Approach to Concept-Level Explanation for Visual Models Without Model Weights

The Interpretability Challenge of Black-Box Visual Models

Deep learning visual models perform impressively on tasks like image classification, segmentation, and caption generation — but most of them are "black boxes" whose internal logic is nearly impossible to inspect. When a model decides a face is "smiling" or identifies "sky" in an image, it's difficult to know which visual features actually drove that decision. This is the core challenge of machine learning interpretability research.

Concept-based attribution methods attempt to explain model behavior using human-understandable concepts, and have attracted considerable attention in recent years. However, existing approaches have several notable shortcomings: they typically only work for classification and regression models, require access to model weights, or depend on large-scale concept-annotated datasets. Most also fail to provide image-level attribution. These limitations make them nearly useless in real-world black-box scenarios.

Among concept-based attribution methods, the most representative early work is TCAV (Testing with Concept Activation Vectors), which trains linear classifiers in the activation space of intermediate model layers to quantify a concept's influence on predictions. Later methods like ACE and ConceptSHAP extended this approach but retained the requirement for internal model access. Running in parallel are pixel-level attribution methods (e.g., LIME, SHAP, Grad-CAM), which are more widely used but explain in terms of pixels or regions rather than semantic concepts familiar to humans (like "beard" or "blue sky"), limiting their readability. The core tension between these two lines of work is that semantically interpretable methods tend to require white-box access, while black-box-friendly methods lack semantic explanatory power. HnS attempts to satisfy both conditions simultaneously — which is precisely what makes it methodologically novel.

The HnS Method: Hide a Concept, Then Track Its Impact

A new paper introduces Hide and Seek (HnS), a concept-level attribution method whose core idea is intuitive from the name itself — first "hide" a concept, then "seek" its effect on the model's output.

HnS's key breakthrough is that it requires neither access to model weights nor a concept-annotated dataset. Given a concept and a set of images, the workflow proceeds roughly as follows:

Four-Step Attribution Pipeline

• Retrieve aligned images: Use a vision-language model (VLM) to retrieve images from the image set that are relevant to the target concept;
• Localize concept regions: Identify the specific regions in each image that correspond to the concept;
• Apply masking: Mask the localized regions;
• Measure attribution: Compare the black-box model's outputs on the original versus masked images — the larger the change, the greater the concept's contribution to the model's decision.

The elegance of this design is that it measures attribution by directly observing changes in model behavior, rather than relying on gradients or activations from internal parameters. This makes the attribution results more faithful — they genuinely reflect how the model actually makes decisions — while naturally supporting local (image-level) attribution.

The "masking" step here is not simply zeroing out pixels; it requires careful design. Common masking strategies include filling with average color, Gaussian noise, or using inpainting algorithms to generate semantically coherent backgrounds. Different fill methods introduce varying degrees of out-of-distribution problems — masked images may fall outside the distribution of the model's training data, causing output changes not because the concept was removed, but because the input itself became "anomalous." This is a fundamental challenge shared by all masking-based attribution methods, and an important factor when evaluating whether masking attribution results are truly faithful. The effectiveness of HnS's attribution depends, in part, on how well its masking implementation controls for distribution shift.

Validation Results: Closer to Ground-Truth Attribution

The research team validated HnS on two datasets: CelebA and COCO-Stuff. The former is a facial attribute dataset; the latter contains rich annotations of objects and background scenes. Together they cover two distinct task types: attribute recognition and complex scene understanding.

Experiments show that compared to existing methods, HnS achieves higher agreement with ground-truth attributions. This means its explanations of "which concept matters" more closely match human-annotated objective answers, rather than being merely self-consistent numerically.

CelebA is a large-scale dataset of approximately 200,000 celebrity face images, each annotated with 40 binary attributes (e.g., "smiling," "wearing glasses," "has beard"), commonly used for face attribute recognition and generative model evaluation. COCO-Stuff extends MS-COCO with pixel-level segmentation annotations for 91 "stuff" categories — amorphous background regions like grass, sky, and water — covering more complex natural scene semantics. Used together, the two datasets enable evaluation of concept attribution methods on both locally-detailed concepts (like facial attributes) and globally-distributed scene semantics, making them a fairly standard evaluation combination in this field.

True Generality: Applicable Across Tasks

One of HnS's most practically valuable features is its ability to explain tasks that existing concept attribution methods simply cannot handle. The paper demonstrates two representative scenarios:

• Image captioning: This type of generative task falls outside the traditional classification or regression paradigm, leaving existing concept attribution methods essentially helpless;
• Satellite image segmentation: Segmentation tasks in remote sensing also lack mature concept explanation tools.

Because HnS treats the model entirely as a black box, observing only input-output relationships, it can in principle be adapted to virtually any visual model, regardless of architecture or task type. This "model-agnostic" property gives it a far broader range of applicability in real-world deployments compared to methods that require white-box access.

Significance and Limitations

HnS represents a pragmatic direction in interpretability research: it acknowledges that in practice we often only have access to a model's API rather than its internal weights, so explanation methods must work under black-box constraints. It uses the intuitive mechanism of masking and observation to sidestep both the reliance on model internals and the need for annotated data.

That said, the method's effectiveness is heavily dependent on two upstream components — the VLM's concept retrieval capability and the accuracy of concept region localization. If the images retrieved by the VLM are poorly aligned with the target concept, or if the localized regions are imprecise, attribution results may be affected. Additionally, the paper currently only provides anonymized code (for peer review) and remains in the academic validation stage; its performance at scale in production settings remains to be seen.

Overall, HnS offers a new path toward faithful, localized concept explanations for black-box visual models — without requiring privileged access — and holds real practical value for applications where models are only accessible via API.