Synthetic Benchmarks Crack the XAI Evaluation Problem: Testing Nine Methods Reveals Significant Limitations

The Core Dilemma of Explainable AI Evaluation

Explainable Artificial Intelligence (XAI) has become a hot research topic in machine learning, yet a long-overlooked problem continues to trouble researchers: how do we know whether an explanation produced by an XAI method is actually correct?

The root of the problem is the lack of a reliable evaluation process — specifically, the absence of so-called "ground truth explanations." When a model is a black box, we don't know which features it truly relies on for decisions internally, making it impossible to verify whether the outputs of an explanation method faithfully reflect the model's reasoning process.

This paper published on arXiv (arXiv:2609.30397v1) targets exactly this pain point, proposing an evaluation framework based on synthetic ground truth — an attempt to provide a more solid foundation for reliably assessing XAI methods.

Why Existing Evaluation Methods Fall Short

The current mainstream approach for evaluating explanation quality is to measure fidelity — how closely an explanation aligns with a black-box model's predictions. In simple terms, it asks how well the explanation can reproduce the model's output.

But the paper sharply points out the flaw in this strategy: fidelity can only quantify "whether an explanation can reproduce the model's output," but cannot guarantee "whether the explanation correctly reflects the underlying decision-making process." There is a fundamental difference between these two things.

The consequences are clear — different explanations may achieve similar fidelity scores while offering contradictory or even misleading interpretations of model behavior. In other words, you might receive two "explanation reports" with nearly identical scores that tell completely different stories. For domains that rely on explanations to make critical decisions — such as healthcare, finance, and law — this is quite dangerous.

Fidelity in XAI typically takes two specific forms: local fidelity measures whether an explanation approximates model behavior near a single sample, while global fidelity measures reproduction across the entire input space. Common calculation approaches involve perturbing the model's output on the subset of "important features" identified by the explanation method and observing the magnitude of change in predictions. Mainstream methods like LIME and SHAP are designed to maximize local fidelity as an optimization objective — which is precisely why they tend to top fidelity leaderboards. This "self-endorsing" evaluation logic is the core problem the paper criticizes.

Building Datasets with "Known Answers" via Controlled Interventions

The paper's core insight is: if ground truth doesn't exist in real-world scenarios, then artificially create it.

The framework relies on controlled interventions to generate synthetic datasets. In such datasets, the importance of each input component is predetermined by design — researchers know from the start which features should be important and which should not.

The elegance of this design is that it allows the constructed ground truth explanations to align directly with the behavior of the model being analyzed. When we understand both the data generation mechanism and can observe model behavior, we have a reliable "measuring stick" to evaluate the performance of various explanation methods.

This synthetic, intervention-based approach fundamentally transforms the evaluation problem from "unverifiable" to "verifiable," establishing a reproducible experimental environment for XAI evaluation.

The concept of controlled interventions draws from the do-calculus in causal inference. During data generation, researchers explicitly manipulate the influence weights of certain feature variables on labels, thereby "encoding" ground truth feature importance at the data level. This aligns with the logic of the "do-operator" in causal models: determining causal relationships between variables through external intervention rather than natural observation, thereby eliminating the influence of confounding variables. Similar ideas appear in robustness evaluation approaches such as "pixel flipping" experiments and "Remove and Retrain" (ROAR), but this framework goes further by embedding ground truth directly into the data generation process rather than relying on post-hoc proxy metrics.

Comprehensive Validation Across Three Data Domains

To ensure the framework's generalizability, the authors instantiated it across three different data domains:

• Binary images: Suitable for intuitively observing the spatial distribution of feature importance
• Tabular data: The most common structured data format in machine learning
• Time series: Complex scenarios involving temporal dependencies

Spanning these three heterogeneous data modalities means the framework is not a "custom solution" tailored to a specific type of problem, but rather capable of comprehensively evaluating explanation methods across diverse environments. This breadth of coverage is critical for validating the universality of any approach.

Test Results Across Nine Mainstream XAI Methods

The paper evaluates nine widely-used XAI methods under this framework. The experimental results send a cautionary signal: current techniques have significant limitations.

The significance of this finding is that many explanation methods performing well on fidelity metrics may expose problems when subjected to rigorous testing against known ground truth. This confirms the earlier judgment — high fidelity does not equal high-quality explanation.

The authors therefore emphasize that synthetic, intervention-based benchmarking holds substantial value for reliably evaluating explanation quality. It's not just a new tool, but represents a paradigm shift in evaluation: moving from "can the explanation reproduce the model's output" to "can the explanation truly explain the model's decision."

The nine XAI methods evaluated in the paper span the most representative technical approaches in the field: gradient-based methods (e.g., Vanilla Gradient, Integrated Gradients), perturbation-based methods (e.g., LIME, SHAP/KernelSHAP), and propagation-based methods (e.g., LRP — Layer-wise Relevance Propagation). These methods operate under very different design assumptions — gradient-based methods assume local model linearity, perturbation-based methods approximate feature contributions through sampling, and propagation-based methods distribute "credit" layer by layer through the network. Because they start from such different premises, their performance differences under the synthetic ground truth benchmark can reveal the applicability boundaries of each method's assumptions across different data types, rather than simply producing a fidelity ranking.

Implications for Industry and Research

This work carries several practical implications for the XAI field.

For researchers, it provides a common measuring stick for comparing different explanation methods side by side, helping to reduce the "everyone measuring differently" chaos in evaluation. For practitioners, it serves as a reminder: don't blindly trust the conclusions of explanation methods — especially in high-risk decision-making contexts, there may be a vast gap between an explanation "looking reasonable" and "actually being correct."

Of course, the synthetic data framework has its inherent limitations — there will always be a gap between artificially constructed data and the complexity of the real world. How to migrate this evaluation approach to real-world business data remains a direction requiring further exploration. But as a controlled experimental environment that provides "known answers," it undoubtedly represents a solid step toward making XAI evaluation more rigorous.