Test-Time Ablation: A Plug-and-Play Method for Improving the Faithfulness of LLM Explanations

Large Language Models (LLMs) are increasingly being deployed in critical decision-making scenarios, where their explanations serve as essential tools for auditing model behavior. However, these explanations often suffer from "unfaithfulness" — they fail to truly reflect the model's actual reasoning process. In the field of Explainable AI (XAI), "faithfulness" is a core evaluation dimension, standing in contrast to "plausibility": plausibility measures whether an explanation "looks reasonable" to humans, while faithfulness measures whether it genuinely reflects the model's internal computations. An explanation can appear perfectly plausible while being entirely unfaithful — like a student writing textbook-perfect solution steps when they actually arrived at the answer by guessing. A recent study from arXiv proposes a test-time optimization method that improves the completeness and reliability of explanations by removing concepts from the input that are not mentioned in the explanation.

Two Dimensions of Unfaithful LLM Explanations: Incompleteness and Unsoundness

The research team clearly distinguishes two core problems with unfaithful LLM explanations: incompleteness and unsoundness.

• Incompleteness refers to explanations that omit factors that actually influenced the answer — like a student who only writes out partial solution steps on an exam.
• Unsoundness refers to explanations that mention factors that didn't truly influence the model's decision — essentially providing fabricated reasons.

Existing improvement methods fall into two main categories: train-time methods typically modify the model's training objective to enhance explanation capabilities — for example, adding faithfulness constraints to the loss function, or using self-consistency training to align the model's reasoning chain with its final answer. Representative work includes Chain-of-Thought fine-tuning and explanation alignment via Reinforcement Learning from Human Feedback (RLHF). These approaches are effective but costly, requiring full model weight access and large-scale GPU computing resources. Test-time methods, on the other hand, intervene on inputs or outputs during the inference phase without modifying the model itself. However, prior test-time methods have mostly focused on the unsoundness problem — for instance, using self-reflection prompting to have the model check whether its explanation contains redundant factors unrelated to the decision. This research fills the gap by directly addressing incompleteness with a test-time approach.

Core Methodology: Concept Removal

The core idea of this method is elegant in its simplicity: after an LLM provides an answer and explanation for a question, the researchers remove concepts from the original input that were not mentioned in the explanation, then re-query the model with this "streamlined" input. This operation eliminates hidden influencing factors that went unacknowledged while preserving the influence of concepts explicitly referenced in the explanation.

The theoretical foundation of this method comes from the counterfactual intervention framework in causal reasoning. In causal inference, the most direct way to determine whether variable X truly affects outcome Y is to remove or alter X while holding other variables constant, then observe whether Y changes. This idea is already widely applied in machine learning interpretability: LIME (Local Interpretable Model-agnostic Explanations) estimates feature importance by perturbing input features, while SHAP (SHapley Additive exPlanations) allocates each feature's contribution to the prediction based on Shapley values from game theory. This research applies the idea in reverse — rather than removing concepts mentioned in the explanation to verify their importance, it removes concepts not mentioned in the explanation to detect whether critical influencing factors have been omitted.

For example, if a model answering "Is this movie review positive or negative?" mentions "excellent acting" but fails to mention "dragging plot," the system removes the plot-related description from the input and re-queries. If the answer changes, it indicates the original explanation was incomplete — it omitted the actual influence of the plot factor.

This process can be performed iteratively, progressively converging toward more faithful explanations.

Experimental Validation: Significant Performance Gains Across Models

The research team conducted rigorous testing on two datasets, covering multiple model families, and employed two independent faithfulness evaluation metrics. Experimental results show that compared to standard prompting methods and faithfulness-encouraging prompting methods, this ablation approach significantly improves explanation faithfulness.

The method's greatest advantage lies in its model-agnostic nature and plug-and-play capability — it can be applied directly during inference without modifying model parameters. Model-agnostic means the method doesn't depend on any specific model architecture or internal mechanism and can be equally applied to GPT-4, Claude, Gemini, LLaMA, and other LLM families. In current industry practice, this characteristic carries significant engineering value: most enterprises access commercial LLM services through APIs with no access to internal model parameters whatsoever; even with open-source models, fine-tuning requires substantial computational and engineering resources. A plug-and-play test-time method can serve as an external "wrapper" that improves explanation quality without altering the underlying model, dramatically lowering the deployment barrier.

This provides tremendous convenience for real-world deployment, particularly suited for:

• API-based scenarios where model weights are inaccessible
• Deployment environments with limited computational resources
• Audit requirements demanding rapid verification of explanation quality

Critical Implications for AI Safety and High-Stakes Decision-Making

In high-risk domains such as medical diagnosis, credit approval, and legal consultation, LLM decision rationales must be transparent and auditable. Incomplete explanations can mask model biases or flawed logic — for example, a model might base decisions on sensitive features like race or gender while deliberately avoiding these factors in its explanation. This phenomenon is known in fairness research as "proxy discrimination": even if a model doesn't directly use protected features on the surface, it may make discriminatory decisions indirectly through highly correlated proxy variables (such as the correlation between zip codes and race, or names and gender). Unfaithful explanations make these hidden biases even harder to detect and correct.

This research provides a practical tool for detecting and reducing such hidden influences, improving the reliability and safety of LLM-assisted decision-making systems. By forcing models to "practice what they preach" — factors mentioned in explanations should genuinely influence decisions, and unmentioned factors should have no influence — this method takes an important step toward building more trustworthy AI systems.

Future Applications and Compliance Value

The method's flexibility gives it broad application potential:

• Enterprise side: It can be directly integrated into existing LLM application workflows, improving explanation quality without retraining models.
• Regulatory side: Regulatory bodies can adopt this technique as an auditing tool to verify whether AI system decisions align with their stated logic.

As AI regulatory requirements grow increasingly stringent, legislative processes worldwide are accelerating. The EU AI Act officially took effect in August 2024 as the world's first comprehensive legal framework regulating AI systems. The act classifies AI systems into four risk categories: unacceptable risk (prohibited), high risk, limited risk, and minimal risk. For high-risk AI systems (including those used in credit assessment, judicial assistance, medical diagnosis, and similar scenarios), the act explicitly requires sufficient transparency and explainability in their decision-making processes, enabling human overseers to understand, challenge, and correct AI outputs. Non-compliant enterprises face fines of up to €35 million or 7% of global annual revenue. Meanwhile, the U.S. AI Executive Order, China's Interim Measures for the Management of Generative AI Services, and other regulations also impose varying degrees of transparency and auditability requirements on AI systems. Against this backdrop, explainability is transforming from an academic research topic into a rigid compliance requirement, and test-time optimization methods like this one will become crucial technical tools for enterprises achieving compliance. The research team's work provides an actionable pathway for taking "trustworthy AI" from theory to practice.