One Repo, Two Loops: Which One Is Actually RL?

The Debate Around Self-Improving Agents: Is This Actually RL?

Whenever someone posts about a "self-improving agent," the comments inevitably fire back: "That's not reinforcement learning — that's just prompt tuning." And half the time, that rebuttal is completely correct.

The core issue is that many projects conflate "searching over configurations" with "actually training parameters." The reef codebase happens to put both loops inside the same repository, which gives us a rare, clean opportunity to draw a clear line: which loop actually touches the model's weights, and which one is just playing word games.

The Harness Loop: The Half That Trains Nothing

The first loop is called the harness loop. Its defining characteristic is straightforward: it only needs a model endpoint — no GPU required.

Its variants — Reefine, SkillClaw, GEPA, Meta-Harness — work by proposing modifications to the harness's own files, then putting the modified code tree on a task list and scoring it against the current version. Whether a change is kept depends on a selection policy; the default rule is that a "net win" is required for a change to stick.

The critical point: nothing moves inside the model's parameters. In the author's own words, this is better described as "search over configurations with a verifier." If someone calls this RL, then the comment section's "that's not RL" rebuttal is entirely warranted — and deserved.

This is actually a valuable reminder. A huge amount of today's AI agent "evolution" is fundamentally about tweaking prompts, tool chain configurations, and scheduling logic — all of which are optimizations at the external framework level that never touch a single numerical value inside the neural network. They may genuinely work, but by technical definition, they are not reinforcement learning.

The Weight Loop: The Half That Actually Moves Parameters

The second loop is the weight recipes side, which includes SAO, OpenClaw^Ad-RL, TTT-Discover, and Guidance-TTT. This side requires GPUs and a full training stack. The author honestly acknowledges that they're only responsible for the harness side and have only read the code on the weight side — that transparency is worth noting.

The most interesting entry here is OpenClaw-RL, because it's specifically designed for served traffic: the policy is updated directly from conversations themselves, with a binary reward signal extracted from the user's next reply.

This is a fairly aggressive online learning setup — the model isn't being trained on an offline dataset but is learning continuously from real conversation streams while serving users.

Online Learning vs. Offline Training is a distinction worth clarifying here. Traditional model training is "offline": collect a dataset, train in batches, then deploy a model with fixed weights. OpenClaw-RL's online learning paradigm means the model continuously receives reward signals and updates its own weights while serving real user requests — deployment and training are the same process. This is enormously challenging from an engineering standpoint: you need to maintain both an inference engine and a training loop simultaneously, while handling consistency issues caused by frequent weight updates. The "binary reward" used here — extracted from the user's next reply — is a proxy for implicit feedback, with logic similar to "user continues the conversation = positive signal." But this also means the reward signal is quite sparse and noisy.

Verified Experimental Results: The GSM8K Homework Flow

One concrete experiment checked into the repository was run with hermes memory disabled:

• A GSM8K math homework flow of 72 sessions
• Qwen3-32B playing a student persona
• Qwen3-4B-Thinking as the policy model being trained
• Rewards provided by a PRM (Process Reward Model) — the frozen base version of that same 4B model, running on its own engine

There's an elegant design choice here: the grader uses the policy model's own starting weights rather than an external, larger model. In other words, the model is being judged by "the self it started as" while evaluating "the self it's becoming," all running on seven GPUs.

Results showed that this run reached the paper's adaptation criterion (three consecutive passing sessions) at session 14. Over the first 36 sessions, the learning curve tracked two metrics — bold rate and list rate — both of which trended downward. That's the full extent of what's been published.

PRM (Process Reward Model) is a scoring mechanism distinct from the Outcome Reward Model (ORM). An ORM gives a single score only on whether the final answer is correct, while a PRM scores each intermediate step in the reasoning process separately, providing a more fine-grained learning signal. This is especially critical for mathematical reasoning tasks: "correct answer but flawed process" and "correct process but a final arithmetic mistake" receive identical scores under an ORM, while a PRM can distinguish between the two. The more notable design choice in this experiment is that the PRM uses the frozen base weights of the policy model (Qwen3-4B-Thinking) rather than a separate external model. The advantage is that this avoids dependence on an expensive large judge model while aligning the scoring standard with the policy model's initial capabilities. The potential limitation is that as the policy model's weights continue to update, the capability distribution between the scorer and the scored may gradually diverge.

The One-Line Test: Which Parameters Moved?

The most practical takeaway from this writeup is a remarkably concise diagnostic criterion. The author closes with a piece of advice:

If a post claims the first loop (the harness loop) is RL, just ask it — which parameters actually moved?

This question is essentially a litmus test for evaluating any "self-evolving AI" claim. The essence of reinforcement learning is updating a policy's parameters via reward signals. If no trainable parameters are being updated, then no matter how much it looks like "self-improvement," it's configuration search or prompt optimization — not RL.

At a time when the Agent concept is being heavily hyped, this kind of disciplined clarity around terminology is especially important. Packaging prompt tuning as RL isn't just a misuse of terms — it actively misleads people's judgment about what a system is truly capable of. By presenting both loops side by side in the same repository, reef makes this boundary impossible to miss.