XAI-Arena: Using Large Language Models to Judge the Quality of Explainable AI Explanations

The Long-Standing Challenge of Evaluating Explainable AI

As machine learning models are increasingly deployed in high-stakes domains like healthcare, finance, and criminal justice, Explainable AI (XAI) has shifted from a nice-to-have to an absolute necessity. Yet one core problem has long plagued the field: how do we judge whether an explanation is actually good?

Traditional XAI evaluation methods rely heavily on subjective human judgment. Researchers typically recruit human participants to rate outputs from different explanation methods — an approach that, while grounded in real user experience, suffers from three fundamental weaknesses: poor reproducibility, difficulty scaling, and a lack of comparability across studies. Because participant pools, rating criteria, and task designs vary so widely between experiments, it's extremely difficult to make fair, apples-to-apples comparisons across XAI methods.

Against this backdrop, a new paper published on arXiv introduces a framework called XAI-Arena, which uses large language models (LLMs) as "judges" to provide a scalable, reproducible approach to evaluating XAI explanation quality.

Inside the XAI-Arena Framework: LLMs as the Referee

The central idea behind XAI-Arena is the "LLM-as-a-judge" paradigm — using large language models to perform comparative evaluations of explanations produced by XAI methods. This approach has already proven effective in evaluating text generation and dialogue systems, and the authors now transfer it to the more specialized domain of explainable AI.

Rather than simply assigning a single score, XAI-Arena emphasizes a multi-dimensional evaluation framework oriented toward different stakeholders. The research team breaks down explanation quality into eight specific dimensions:

• Perceived simplicity: Is the explanation sufficiently concise?
• Clarity: Is it clear and easy to understand?
• Task adequacy: Does it fit the specific application context?
• Trust calibration: Does it help users develop an appropriate level of trust?
• Actionability: Can the explanation guide real-world decisions?
• Transparency: How much does it expose the model's internal logic?
• Faithfulness: Does it accurately reflect the model's actual decision-making process?
• Overall interpretability: A holistic evaluation

This eight-dimensional design is notably thoughtful. It moves beyond one-dimensional thinking — whether "simpler is better" or "more detail is better" — and acknowledges that different contexts and audiences have fundamentally different needs from explanations.

Multi-Perspective Evaluation Across Different Roles

Another key feature of XAI-Arena is the introduction of "stakeholder personas." The same explanation can carry entirely different value for a data scientist, a domain expert, and an everyday end user. A feature importance plot full of technical terminology might be useful to an engineer but completely meaningless to a patient or a loan applicant.

XAI-Arena instructs the LLM to adopt different stakeholder perspectives during evaluation, thereby capturing this "relativity of explanation quality." This design makes evaluation results far more reflective of the diverse needs found in the real world.

Background: The Rise of LLM-as-a-Judge

This paradigm has its theoretical roots in a series of studies from around 2023, with seminal work including Zheng et al.'s MT-Bench/Chatbot Arena, which found that GPT-4 as a judge agreed with human preferences over 80% of the time. The core assumption is that large language models, after being aligned on massive amounts of human feedback, have internalized a significant degree of human aesthetic and quality judgment standards, making them capable of substituting for human scoring in certain tasks. Key limitations of this paradigm have also been widely discussed: LLM judges exhibit "position bias" (favoring whichever option is presented first), "verbosity bias" (preferring longer responses), and "self-preference bias" (rating outputs from their own model family more highly). When XAI-Arena applies this paradigm to explainable AI, it must additionally navigate the domain knowledge threshold required for judging explanation quality — making the design more challenging than general text evaluation.

Large-Scale Benchmarking: The Advantage of Scale

Building on this framework, the research team conducted systematic benchmarking. They compared different XAI explanation methods across multiple datasets, multiple machine learning models, and multiple stakeholder personas.

This kind of large-scale evaluation is precisely where LLM judges hold their biggest advantage over human evaluation. Conducting assessments across so many combinations with human participants would be enormously time-consuming and costly, with no guarantee of consistency. LLM judges, by contrast, can be highly standardized, run in batches, and reproduced at any time — directly addressing the three core weaknesses of traditional methods.

Through such benchmarking, XAI-Arena can reveal systematic quality differences among XAI methods, giving researchers data-driven support for choosing the right explanation approach.

Background: Major Explanation Methods in XAI

The mainstream explanation methods in XAI today include feature-importance-based approaches like SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-agnostic Explanations). SHAP originates from Shapley values in game theory and calculates each feature's marginal contribution to model predictions; LIME generates local explanations by fitting an interpretable linear surrogate model around a given sample. Other approaches include attention-based methods, counterfactual explanations ("how would the prediction change if feature X varied?"), and rule-based explanation systems. These methods each involve trade-offs in interpretability, computational cost, and applicable model types — and the lack of a unified benchmark for cross-scenario comparisons is precisely the gap XAI-Arena aims to fill.

Human Validation: How Well Do LLM Scores Align with Human Ratings?

The biggest question about using LLMs as judges is: can their judgment be trusted? Could their assessments be wildly at odds with what real humans actually think?

To answer this critical question, the research team conducted human validation experiments, analyzing the correlation between LLM-generated scores and human ratings. The results showed a strong positive correlation (Spearman's rho = .693, p < .001).

A Spearman correlation coefficient of approximately 0.69 is considered quite solid in social science and evaluation research — it indicates that the LLM's judgments align strongly with human judgments in their overall trends, while remaining statistically significant. This means the LLM judge isn't "scoring randomly" but is genuinely capturing the explanation quality characteristics that humans care about.

That said, a correlation of 0.69 warrants an honest reading: it indicates "strong correlation" but certainly not "perfect equivalence." LLM judges can serve as highly efficient tools for large-scale screening and comparison, but in final high-stakes decision-making scenarios, human judgment cannot be fully replaced.

Background: Understanding Spearman's Rho

Spearman's rho is a non-parametric statistic that measures the monotonic correlation between two sets of rankings, with values ranging from -1 to 1. Unlike the Pearson correlation coefficient, it does not require the data to follow a normal distribution, making it better suited for ordinal data like subjective ratings. In behavioral science and evaluation research, rho > 0.6 is generally considered "strong correlation," while rho > 0.8 is "very strong correlation." The rho = 0.693 reported in this study means: when the LLM considers explanation A better than explanation B, human raters give the same ranking approximately 69% of the time. p < .001 indicates the result is statistically highly significant, effectively ruling out chance. It's worth noting that correlation coefficients measure "consistency in ranking trends" rather than "agreement in absolute scores" — so there may still be systematic differences between LLM and human scores on specific numerical values.

Significance and Outlook: The Future of Standardized XAI Evaluation

XAI-Arena's value lies not only in providing a concrete evaluation tool, but in exploring a viable path toward standardizing and automating explainable AI evaluation.

For the XAI research community, this framework has the potential to alleviate the long-standing problem of incommensurability between studies. When different teams can use the same reproducible LLM judge system to evaluate their methods, the comparability of academic results will be greatly enhanced.

From a broader perspective, this is another example of the "LLM-as-a-judge" paradigm penetrating specialized vertical domains. From evaluating conversation quality to evaluating code, to now evaluating "explanations of explanations," large models are increasingly taking on evaluative roles. This reflects a trend worth pondering: AI is not only generating content, it is also assessing the quality of content.

Of course, many questions remain open in this direction — how LLMs' own biases affect their judgments, whether different base models as judges would reach different conclusions, and the reliability of LLM judgment in highly specialized domains, among others. XAI-Arena makes a strong start in this direction, but making "AI evaluating AI" a recognized academic standard will require a much longer journey.