Explanatory World Models (EWM): Solving Reinforcement Learning's Delayed Reward Problem with Causal Explanations

The Credit Assignment Dilemma in Reinforcement Learning

In generative recommender systems, reinforcement learning (RL) is increasingly used to optimize outcomes that may only manifest after hundreds or thousands of interactions. A user's click today might not translate into a genuinely valuable outcome until much later. This gives rise to one of RL's most persistent core challenges — credit assignment.

Standard RL algorithms propagate rewards based on temporal proximity rather than explicit causal relevance. In other words, actions closer in time to when a reward occurs receive more credit, while early actions that were truly decisive but temporally distant receive severely attenuated learning signals. This mechanism directly causes poor sample efficiency: models require massive amounts of interaction data just to figure out which early decisions actually mattered.

Credit assignment has a long research history in RL. Classical Temporal Difference (TD) learning propagates value estimates forward incrementally via bootstrapping, while policy gradient methods (such as PPO) rely on a discount factor γ to exponentially decay future rewards — a reward t steps away retains only γ^t of its weight. Both mainstream mechanisms share an implicit assumption: temporal proximity is a proxy for causal relevance. In long-horizon tasks, however, this assumption systematically breaks down. For instance, content recommended at step 3 in a recommender system may directly influence a user's retention decision at step 200, yet after 197 steps of γ^197 decay, the TD error's learning signal has essentially vanished. This problem is especially acute in sparse reward settings — where an entire episode receives only a single reward at the very end — forcing models to rely on massive random exploration to stumble upon effective action sequences.

EWM: Turning Explanations into Learning Signals

To address this pain point, researchers proposed Explanatory World Models (EWM) — an attribution framework whose core idea is to use "explanations" to identify early events that likely contributed to an eventually realized outcome.

Rather than mechanically decaying rewards over time, EWM attempts to answer a more fundamental question: which earlier events meaningfully contributed to the current outcome? The research team instantiates EWM using a frontier language model, using the explanations it generates directly as learning signals.

The technical key here is a "conservative hierarchical operator." It recursively redistributes rewards, tracing credit back from the outcome moment to relevant earlier actions, while preserving the undiscounted cumulative return of the entire episode. This constraint is critical — it ensures that reward redistribution merely "moves" credit along the time axis rather than creating or erasing total reward, thereby maintaining the consistency of the original problem.

Three Explanation Variants: What Makes an Explanation Useful?

Perhaps the most intellectually rich aspect of the EWM framework is the researchers' exploration of the philosophical question: what constitutes a useful explanation? Rather than staying at the engineering level, they drew inspiration from causal and philosophy-of-science literature to operationalize three complementary explanation variants.

Actual Causation

Inspired by Halpern's theory of actual causation, this variant focuses on which events genuinely served as causes of the outcome in the situation that actually unfolded, emphasizing causal contribution in a counterfactual sense.

Halpern and Pearl's theory of actual causation formalizes the intuition that "X caused Y in a specific context." Its core tools are Structural Causal Models (SCM) and counterfactual reasoning: if, in the actual situation, intervening to prevent X from occurring also prevents Y, then X is an actual cause of Y. This is fundamentally different from statistical correlation — two variables can be highly correlated without being causally related. Bringing this framework to credit assignment means the system no longer asks "which action was closest to the reward" but rather "if this action were undone, would the reward still occur?" This counterfactual perspective naturally aligns with RL's exploration needs, though its computational complexity is high — a key motivation for EWM's use of language models for approximate inference.

Explanatory Simplicity

Inspired by Lombrozo's research on explanatory simplicity, this variant favors simpler, more parsimonious explanations. Cognitive science research suggests that humans tend to regard simpler explanations as more likely to be true — a principle imported into reward attribution.

Hard-to-Vary Explanations

Inspired by physicist David Deutsch's notion of "hard-to-vary" explanations, this variant emphasizes that good explanations should be tightly coupled — their constituent parts so interdependent that they cannot be arbitrarily altered without undermining the explanation's validity.

These three perspectives approach the concept of "high-quality explanation" from the angles of causality, cognitive economy, and explanatory rigidity, respectively, forming a pluralistic understanding of what makes an explanation valuable.

In his book The Beginning of Infinity, David Deutsch argues that the key to scientific progress is producing explanations that are "hard to vary" — a good explanation must be tightly coupled in its details, such that modifying any detail breaks the entire explanation. He uses Newton's law of universal gravitation as an example: changing the law to "Earth attracts apples but not the Moon" would destroy its ability to explain tides, planetary orbits, and other phenomena — the tight constraints between parts make arbitrary modification costly. This complements Popperian falsifiability: hard-to-vary explanations are not only falsifiable but also structurally resistant to post-hoc "patching" to evade falsification. In the context of credit assignment, this principle means preferring explanations that involve tightly logical connections between actions, rather than loose narratives where actions can be freely added or removed without affecting the explanation's force.

Multi-Benchmark Validation: Greater Delays, Greater Gains

The research team evaluated EWM across four distinct settings, spanning the full spectrum from controlled experiments to industrial-scale deployment.

Controlled Long-Horizon RL Benchmarks (Crafter and WordCraft): On these two long-horizon RL benchmarks, EWM significantly outperformed both PPO and LaRe. More critically, the performance gains were largest when reward delay was most severe — directly validating EWM's design rationale.

Public Sequential Recommendation Benchmark (KuaiRand): On this real-world recommendation dataset, EWM outperformed both SFT (supervised fine-tuning) and LaRe in offline settings, and surpassed A2C in online settings.

Industrial-Scale Generative Recommender System: In an offline evaluation of an industrial-scale generative recommender system, EWM substantially improved Recall and enhanced multiple behavioral proxy metrics — demonstrating that EWM is not only effective on academic benchmarks but also has real potential for large-scale production deployment.

Crafter is a 2D survival game benchmark inspired by Minecraft, designed specifically to test long-horizon RL. It requires agents to simultaneously manage health, hunger, resource gathering, and technology tree unlocks, with highly sparse and severely delayed rewards. WordCraft is a text-based combinatorial reasoning game in which agents must combine elements into target items through multi-step operations, similarly emphasizing causal chains across time steps. LaRe (Language-Reward) is the most directly comparable baseline to EWM — it also uses language models to assist with reward shaping, but through direct reward relabeling rather than EWM's conservative hierarchical operator, which redistributes rewards while strictly preserving total episode return. This makes EWM theoretically less prone to introducing reward hacking. The finding that EWM's advantage is greatest under the most severe reward delays provides direct empirical support for its design motivation.

Significance and Outlook

The most inspiring aspect of EWM is how it transforms "explanation" from a post-hoc interpretability tool into a proactive, structured, and reusable learning signal. Traditional RL treats credit assignment as a purely numerical optimization problem, while EWM introduces language model reasoning capabilities and the theoretical framework of causal philosophy to provide principled judgments about which actions matter.

The results demonstrate that explanations can provide structured and reusable credit for learning from delayed outcomes. As the reasoning capabilities of large language models continue to advance, using them to assist or even reshape the core mechanisms of reinforcement learning may become an increasingly important direction. For real-world applications like generative recommendation — where rewards are highly delayed — this approach holds particular practical value.

That said, the framework raises new questions: How reliable are the explanations that language models produce? How can explanation quality be quantitatively evaluated? What are the boundary conditions for different explanation variants across different tasks? These are questions that future research will need to address.