Demystifying Self-Discovered RL: Is Learning History an Asset or a Burden?

From RSI to the Era of Experience

The road toward recursive self-improvement (RSI) and general intelligence has long been split between two approaches: macro-level scaling of language models on one side, and the interaction-driven principles championed by the "Era of Experience" on the other. But regardless of architecture, any self-improving system ultimately rests on its underlying optimization engine. A newly published arXiv paper (arXiv:2609.35897v1) poses a sharp question: if general intelligence truly requires learning from grounded interactions, do reinforcement learning (RL) update rules themselves possess the capacity for cumulative adaptation?

From Black-Box Algorithms to Mechanistic Auditing

Algorithm self-discovery has made remarkable progress in recent years. A standout example, Disco103, has surpassed the classic PPO algorithm and achieved state-of-the-art performance across multiple benchmarks. Yet researchers point out a long-neglected problem: the internal update mechanics of these self-discovered rules remain an unexamined black box. We know they work — we just don't know why they work, or under what conditions they might fail.

The paper's contribution is to provide the first causal mechanistic audit of self-discovered RL rules. The research framework is organized directly around the five pillars of the "Era of Experience": extended time horizons, grounded reward scales, continuing data streams, within-lifetime change, and exploration depth.

Algorithm Self-Discovery is a branch of meta-learning in which the optimization algorithm itself becomes the object of learning — the system automatically searches for and generates update rules more efficient than those hand-designed by humans, through interaction with an environment. Disco103 is a leading example of this direction: via evolutionary search or gradient-based meta-learning, it discovers a general-purpose parameter update formula across a large number of tasks, outperforming Proximal Policy Optimization (PPO) on multiple standard RL benchmarks. PPO is the most widely used policy gradient algorithm in both industry and academia, renowned for its training stability — making Disco103's improvement a significant benchmark achievement. The advantage of self-discovered algorithms is that they escape the constraints of human inductive bias, but this also means their internal logic is nearly opaque to researchers. The algorithm "knows what to do," but no one knows "what it is actually doing" — and that is precisely the motivation for causal mechanistic auditing.

Core Methodology: Surgical State Interventions

The most distinctive aspect of this work is its experimental methodology. The researchers precisely test when learning history acts as an asset versus a burden by pinning, freezing, and transplanting recurrent states while keeping meta-parameters fixed.

This "surgical" intervention approach is essentially a causal inference experimental design. Rather than simply observing an algorithm's input-output behavior, directly manipulating internal states isolates the independent contribution of each factor, avoiding confounding variables. This is precisely what allows the paper's central question — "is learning history an asset or a burden?" — to be answered rigorously.

Recurrent state in this context refers to a hidden vector that carries historical information, analogous to the hidden state in a recurrent neural network (RNN). Self-discovered RL rules typically use memory-capable network architectures (such as LSTM or gated RNNs) as "meta-controllers" that receive the current reward, observation, and previous action at each timestep, then output a parameter update. This hidden vector accumulates the "learning history" the agent has distilled from past interactions. The pinning operation fixes the vector to a specific value (e.g., a zero vector or one from a different training phase), cutting off the inflow of historical information. Freezing preserves the vector's value but prevents further updates. Transplanting injects the historical state from one task into the running state of another. Together, these three operations form a controllable causal intervention toolkit, enabling researchers to separately quantify the effects of historical content versus the mechanism that maintains it.

Three Key Findings

The audit produced three organizing findings that define the boundaries of learning history's role.

Recurrent History Expands the Available Reward Scale

The first finding shows that recurrent history actively expands the usable reward scale. When history is preserved, the system maintains a reward window spanning six orders of magnitude. Under zero-pinning conditions, that window shrinks to just three orders of magnitude. This demonstrates that learning history plays a substantive role in handling reward signals across widely varying scales.

The Penalty from Mismatched History Stems from "Perpetual Clamping"

The second finding is more nuanced. By decoupling historical content from its maintenance mechanism, researchers found that the performance penalty from mismatched history does not originate from the historical content itself — it comes from perpetual clamping. In other words, if transplanted states are allowed to evolve naturally, the burden diminishes substantially. This suggests the problem is often not "what was remembered" but "how that memory is maintained."

Apparent Adaptation Advantages Under Environmental Change May Be Illusory

The third finding carries a cautionary message. Under environmental non-stationarity, controlling for replay retention reverses the apparent adaptation advantage over DQN. This reveals that external data turnover may be confounding internal plasticity — what appears to be a stronger learned adaptation capacity within the algorithm may, in reality, be an artifact of data flow rather than genuine internal flexibility. This finding raises serious methodological demands on evaluating the true capabilities of self-evolving algorithms.

Replay retention refers to the policy of retaining historical data in the experience replay buffer, a mechanism that offline algorithms like DQN rely on to repeatedly sample and learn from past experiences. In non-stationary environments — where reward distributions or dynamics change over time — the ratio of new to old data in the replay buffer has a critical impact on performance: retaining too much old data can cause the algorithm to "live in the past," while clearing the buffer discards valuable prior experience. This paper finds that when researchers control for replay retention ratios, the self-discovered rule's adaptation advantage over DQN disappears — suggesting the previously observed advantage may not reflect an internally evolved plasticity mechanism, but simply the difference in default data-freshness management configurations between the two algorithms. This finding has broad methodological implications for comparative evaluation of RL algorithms in non-stationary environments.

Implications for Next-Generation Self-Evolving RL

These conclusions do not hold only for a single rule. The research validated findings through capability thresholds and transferred the audit methodology to a second self-discovered rule, OPEN, demonstrating the approach's generalizability.

The deeper value of this work lies in re-anchoring the macro-level ambition of recursive self-improvement to the micro-level dynamics of learning. The field loves to discuss model scale, emergent capabilities, and grand narratives of self-improvement — but if the underlying update rules harbor poorly understood vulnerabilities, those macro-level goals lack a reliable foundation.

The paper accordingly calls for establishing a set of foundational audit standards for next-generation self-evolving RL algorithms. This is not only a test of specific results like Disco103, but a methodological grounding for the entire research paradigm of "self-discovered optimization engines." When algorithms begin discovering algorithms, we need — more than ever — tools that can open the black box and verify causality.