ReShap: Making Deep Reinforcement Learning Interpretable in High-Dimensional, Multi-Action Scenarios

ReShap extends Shapley interpretability to high-dimensional, multi-action reinforcement learning via feature extraction, SHAP attribution, and policy distillation.
ReShap is an interpretability framework for deep reinforcement learning policies that overcomes the limitations of prior Shapley-value methods restricted to low-dimensional states, two actions, and a single linear surrogate. It extracts compact features from image observations, applies SHAP attribution, and distills the deep policy into interpretable surrogates—decision trees, linear regression, or logistic regression—under the policy's visitation distribution. Evaluated on Atari and Box2D with three deep RL algorithms plus a human subject study, ReShap maintains competitive task performance while decision tree surrogates generally outperform linear and logistic regression in fidelity and return, with linear regression weakest. This shows Shapley-based interpretability can extend to high-dimensional, multi-action scenarios without substantially sacrificing performance.
ReShap: Making Deep Reinforcement Learning Interpretable in High-Dimensional, Multi-Action Scenarios
Deep reinforcement learning (RL) excels at complex tasks, but its policies are driven by deep neural networks, making the decision process a black box. To make RL policies more trustworthy, researchers have tried to explain policy behavior using game-theoretic tools such as Shapley values. However, existing work (such as the SILVER framework) is typically limited to low-dimensional state spaces, two-action settings, and a single linear surrogate model. The recently proposed ReShap framework attempts to break these limitations by extending Shapley-based interpretability to high-dimensional, multi-action environments.

How ReShap Works: From Images to Interpretable Surrogates
ReShap is a policy distillation framework. It first extracts a compact feature representation from raw image observations, then performs SHAP-based attribution analysis on these features to characterize each feature's contribution to policy decisions. Next, ReShap distills the deep RL policy into an interpretable surrogate model, which is trained on all (state, policy action) pairs under the policy's visitation distribution.
The key to this pipeline is that the surrogate model must not only faithfully reflect the original policy's behavior but also remain interpretable. ReShap instantiates three families of interpretable models—decision trees, linear regression, and logistic regression—and compares their fidelity and task return.
Shapley values originate from cooperative game theory, and their core idea is to fairly distribute the total payoff of a "coalition" among its participants: each participant's contribution equals the average marginal gain it brings when joining across all possible subsets. In machine learning interpretability, SHAP (SHapley Additive exPlanations) treats features as participants and the model output as the total payoff, thereby assigning each feature an attribution value that satisfies the axioms of local accuracy, missingness, and consistency. A key property of SHAP is additivity: the sum of all features' SHAP values equals the difference between the model output and a baseline (usually the average prediction), which allows explanations to be decomposed exactly. However, computing exact Shapley values requires enumerating all feature subsets, with complexity growing exponentially in the number of features, so practical implementations typically adopt variants such as Kernel SHAP, Tree SHAP, or sampling approximations. ReShap leverages SHAP to attribute the extracted compact features, thereby characterizing the basis of policy decisions at the feature level.
Policy distillation is a form of knowledge transfer, typically referring to having a smaller "student" model imitate the behavior of a larger "teacher" model. In reinforcement learning, the supervision signal for distillation can be either the action probability distribution output by the teacher policy (soft labels) or the actions the teacher actually takes (hard labels). ReShap adopts the latter: it collects (state, policy action) pairs under the policy's visitation distribution, uses the actions selected by the deep RL policy at these states as labels, and trains an interpretable surrogate model to reproduce this mapping. Here, the "policy visitation distribution" refers to the distribution of states the agent actually encounters when running under the original policy, which determines the training coverage of the surrogate model. Fidelity measures the degree to which the surrogate model's decisions agree with the original policy on the same states and is the core metric for evaluating distillation quality; high fidelity means the surrogate model can accurately substitute for the original policy, making its explanations more trustworthy.
Experimental Setup and Multi-Source Validation
ReShap is evaluated in Atari and Box2D environments using three deep RL algorithms. According to the paper, the study also includes a human subject study to assess the clarity and trustworthiness of the generated policies. Cross-validation of multiple sources shows that ReShap maintains competitive task performance while decision trees generally outperform linear and logistic regression, with linear regression performing weakest in most settings.

This conclusion is consistent across different algorithms and environments, indicating that decision trees as surrogate models achieve a good balance between fidelity and interpretability.
The Atari 2600 benchmark is one of the most commonly used visual-input testbeds for deep reinforcement learning, comprising dozens of games with raw pixel observations, action spaces that are typically discrete combinations of multiple buttons (e.g., Pong has 6 actions), and high-dimensional states with partial observability. Box2D, on the other hand, is a collection of continuous-control environments based on a physics engine (such as LunarLander and BipedalWalker), with low-dimensional continuous vector states and continuous-valued actions, commonly used to test algorithm performance in continuous action spaces. Together, the two cover two typical scenarios—high-dimensional visual input and continuous control—and correspond to two action settings: discrete multi-action and continuous action. This is precisely why ReShap's validation on these two types of environments and three deep RL algorithms can support its conclusion of "extending to high-dimensional, multi-action scenarios" rather than applying only to a single setting.
The Trade-off Between Performance and Interpretability
ReShap's core contribution is demonstrating that Shapley-based interpretability can be extended to high-dimensional, multi-action scenarios without sacrificing too much performance. Decision tree surrogates perform best in most cases, likely because tree models can better capture nonlinear decision boundaries while remaining readable. The weakness of linear regression reminds us that simple linear models may be insufficient to approximate complex policies.
Impact on Human Trust
The human subject study aims to assess the clarity and trustworthiness of policies. Although the original paper does not disclose specific results, this component emphasizes the ultimate goal of interpretability research: enabling human users to understand and trust RL systems. By providing feature attributions and interpretable surrogates, ReShap takes a step toward building trustworthy RL.
In interpretable AI research, human subject studies typically use controlled experiments: participants observe the behavior or explanations of the original policy and the interpretable surrogate, and then their understanding, trust, and willingness to accept the system are measured through questionnaires or tasks. Common metrics include clarity ratings of explanations, predictive accuracy of policy decisions (i.e., whether participants can anticipate the agent's actions based on explanations), and subjective trust scales. The significance of such studies lies in distinguishing "technically interpretable" from "humanly understandable": even if a surrogate model has high fidelity, its explanations may not enhance trust if their form does not align with human cognitive habits. Therefore, human evaluation is a key step in moving interpretability research from algorithmic metrics to practical application, which also explains why ReShap introduces a subject study beyond quantitative experiments.
Limitations and Future Directions
ReShap currently still relies on SHAP attributions, which may incur high computational costs. In addition, the balance between surrogate fidelity and interpretability仍需 further exploration. Future work could explore more efficient feature attribution methods or extend the framework to more complex action spaces and partially observable environments.
Overall, ReShap provides a practical framework for interpretable reinforcement learning in high-dimensional, multi-action scenarios and validates the effectiveness of decision tree surrogates through multi-environment experiments.
One-line Summary
ReShap extends Shapley-based interpretability to high-dimensional, multi-action reinforcement learning through feature extraction, SHAP attribution, and policy distillation.
Paragraph Summary
ReShap is an interpretability framework for deep reinforcement learning policies, designed to overcome the limitations of previous Shapley-value-based methods that apply only to low-dimensional states, two actions, and a single linear surrogate model. It first extracts compact features from image observations, uses SHAP to attribute the features, and then distills the deep policy into interpretable surrogate models such as decision trees, linear regression, or logistic regression under the policy's visitation distribution. The study evaluates on Atari and Box2D environments with three deep RL algorithms, supplemented by a human subject study. Results show that ReShap maintains competitive task performance while decision tree surrogates generally outperform linear and logistic regression in fidelity and return, with linear regression performing weakest. This demonstrates that Shapley-based interpretability can be extended to high-dimensional, multi-action scenarios without substantially sacrificing performance.
Related articles

RePlan-VLA: Progress-Aware Replanning Breakthrough for Long-Horizon Robotic Manipulation
The RePlan-VLA framework converts VLA models' short-horizon capabilities into reliable long-horizon execution through progress-aware adaptive replanning, improving success rates by 21.9% and 10.4% on RoboCasa365 and CALVIN, and validating real-world deployment on a Franka Research 3 robot.

Geometric Bridge Flow Matching: A New Breakthrough in Blind Rigid Motion Correction
Geometric bridge flow matching enables blind rigid motion correction in head CBCT, achieving 37.1 dB PSNR and 0.27 mm RPE on simulated data in ~9.1 minutes, outperforming existing methods.

Google Releases Gemini 4 Hydrogen: 2.5x Faster Without Sacrificing Intelligence
Google releases the lightweight Gemini 4 Hydrogen: 2.5x faster task completion, AA Intelligence Index of 54 beating its flagship sibling, 7.3s single-step on Agent Arena faster than Claude and GPT rivals, four thinking levels with pay-as-you-need pricing, now available across AI Studio and more.