Diversity-Aware Experience Replay via Statistical Leverage Scores

In off-policy deep reinforcement learning, the experience replay buffer is typically dominated by large amounts of routine operation data, while rare events that define task boundaries are scarce. To address this, researchers propose a diversity-aware experience replay method based on statistical leverage scores, which identifies and prioritizes sampling of "unusual" transitions in the buffer to improve success rates on rare tasks.

Core Problem: The "Routine-ness" Bias of Replay Buffers

In physical robot scenarios, the vast majority of transitions in the replay buffer come from normal operation, and these transitions are highly similar to each other. However, what truly determines task success or failure is often those rare edge cases—for example, key decision points when a robotic arm pushes an object near an obstacle. Traditional uniform sampling drowns out these rare events, causing the learning signal to be biased toward routine behavior.

Researchers point out that each stored transition can be measured by its statistical leverage score to gauge how unusual it is relative to the buffer's own data. This score is computed solely from the stored numerical values, requiring no labels, rewards, or value network.

Experience replay buffer is a core component in off-policy reinforcement learning. It stores transitions (state, action, reward, next state) generated by the agent's interaction with the environment, and samples mini-batches from it during training to update the policy or value function. This approach breaks the temporal correlation between samples and improves data utilization, but it also introduces a sampling bias problem: if one type of transition overwhelmingly dominates the buffer, uniform sampling will concentrate the learning signal excessively on these routine behaviors, ignoring rare events that determine task success or failure. In practical scenarios such as robotic manipulation, this bias is particularly pronounced because safe, smooth routine operation data naturally constitutes the vast majority.

Method Design: Leverage Scores and Mixed Sampling

Each mini-batch is drawn from a mixture distribution of this score and uniform sampling. Specifically, the leverage score measures the degree to which an individual transition is anomalous along the direction of greatest data variation. However, direct application encounters two key challenges, and researchers provide solutions for each.

Challenge 1: Direction Selection under One-Hot Encoding

When actions are stored as one-hot indicators, constructing scores along the direction of greatest data variation fails because indicator columns always win in the variance ranking. The researchers' strategy is to select feature columns with the heaviest tails (maximum kurtosis), which can reveal small, outlier groups far from the main body.

One-hot encoding is a common way to represent discrete actions in reinforcement learning, for example encoding three actions "move left, move right, push forward" as [1,0,0], [0,1,0], [0,0,1] respectively. Under this representation, each feature column takes values of either 0 or 1, and its variance is directly determined by action frequency: columns corresponding to frequent actions have large variance, while those corresponding to rare actions have small variance. Therefore, if leverage scores are computed along the direction of greatest data variation (i.e., the feature column with the largest variance), the indicator columns of frequent actions will always win, causing the score to fail to identify truly rare transitions. Researchers instead select the feature column with the largest kurtosis. Kurtosis measures the heaviness of a distribution's tails; a heavy-tailed distribution means there exist a small number of outliers far from the main body, which exactly corresponds to rare but critical action or state combinations.

Challenge 2: The Cancellation Effect of Importance Weights

Prioritized experience replay typically uses importance weights to keep gradients unbiased. However, research finds that this correction completely cancels the method's effect here: under full correction, training performance is no different from uniform sampling. Therefore, researchers choose to abandon the correction and deliberately optimize a rebalanced objective.

Importance sampling is a classical technique used in Prioritized Experience Replay (PER) to correct sampling bias. When sampling from the buffer with non-uniform probabilities, each sample's gradient needs to be multiplied by an importance weight (typically the inverse of the sampling probability) to ensure the expected gradient matches that of uniform sampling, thereby maintaining unbiased updates. However, this correction introduces a fundamental contradiction in diversity-aware replay: if the sampling bias is fully corrected, the effect of prioritizing rare transitions is mathematically canceled out, and the training dynamics degenerate to be indistinguishable from uniform sampling. Researchers therefore choose to abandon importance weight correction, accept biased gradient estimates, and instead directly optimize a rebalanced objective function—giving rare transitions greater weight in the loss to guide the policy toward edge cases that determine task success or failure.

Windowed Handling of Relational Rare Events

When rare situations in the log are relational—manifesting only through the relationship of a segment of behavior to its goal, rather than any single stored coordinate—per-transition scores fail. Researchers apply the same tail rule to short trajectory windows relative to the goal description, thereby recovering the signal.

On a three-joint robotic arm task where the object must be pushed around obstacles, the method improves the success rate on rare tasks from 0.49 to 0.69, while the labeled oracle upper bound is 0.74. This result demonstrates the method's potential in real robot scenarios.

Practical Implications and Limitations

The method's advantage lies in being fully data-driven, requiring no additional labels or reward signals, with low computational cost. However, it relies on the selection of stored feature columns, and may require further adaptation for high-dimensional perceptual inputs. Additionally, abandoning importance weight correction means the optimization objective shifts, which may require tuning the mixing ratio across different tasks.

Overall, this research provides a concise and effective statistical tool for diversity sampling in experience replay, particularly suitable for robot learning scenarios where rare events dominate task performance.

One-line Summary

Using statistical leverage scores to prioritize sampling of rare transitions improves success rates in robotic reinforcement learning tasks.

Paragraph Summary

This article addresses the problem in off-policy deep reinforcement learning where experience replay buffers are dominated by routine operation data and rare events are drowned out. It proposes a diversity-aware experience replay method based on statistical leverage scores. The method computes each sample's unusualness solely from stored transition values, requiring no labels, rewards, or value network, and mixes this score with uniform sampling to form the sampling distribution. To address the failure of variance ranking under one-hot encoding, researchers select heavy-tailed feature columns with maximum kurtosis to construct scores; to address the cancellation of prioritized sampling effects by importance weight correction, researchers abandon the correction and directly optimize a rebalanced objective. For relational rare events, trajectory windowing is used to recover the signal. On a three-joint robotic arm obstacle-avoidance pushing task, the success rate on rare tasks improves from 0.49 to 0.69, approaching the oracle upper bound of 0.74.