Fixed-Task-Set RL vs. Continuous Online-Traffic RL: The Battle Over Agent Training Approaches

An Overlooked Question: After the Model Ships

Most discussions about training agents with reinforcement learning focus on what happens before release: how to design the task set, how to define rewards, how long to train. But a practitioner on Reddit raised a more practical question—what happens after release?

He used Microsoft's 4B coding agent as an example. This model was trained almost entirely through RL, without distillation from a larger teacher model—essentially a well-executed offline RL pipeline: fixed task set, reward function, one training run, one release. At the moment of release, the model's performance on the test set was determined. But once it actually went into production, the tasks it failed on were precisely the ones nobody had put into the task set.

This question leads to two fundamentally different training philosophies: retraining on a fixed set on a schedule, or continuously learning from the traffic the agent has actually served since its last update?

Where Does Offline RL Hit Its Limits

Microsoft's 4B coding agent case is worth examining closely. It demonstrated that a small model can achieve usable coding ability through pure RL, without relying on distillation from a large model. This in itself is a powerful validation of the offline RL methodology.

But the implicit assumption of offline RL is that the task set represents the real distribution. Once deployed, that assumption starts to erode. User requests are wildly varied, and failure modes far exceed what the task set designers imagined. If you only retrain periodically on a fixed set, the model is forever chasing a distribution it has never seen.

This isn't to say offline RL has no value. The advantage of a fixed task set is clear scoring, controllable processes, and reproducible results. For scenarios requiring strict auditing, this may be the only acceptable approach. The problem is that when an agent exists as a continuously running service, the gap between the offline loop and real demands only grows wider.

Offline reinforcement learning (Offline RL) is a class of RL methods that train a policy solely on a pre-collected fixed dataset without further interaction with the environment during training. In contrast, online RL samples new data from the environment while training. The core advantage of offline RL is that the training process is fully reproducible and auditable: fixed dataset, fixed reward function, fixed number of training rounds, hence stable results and easy regression testing. But it has a well-known theoretical difficulty—distribution shift: the policy tends to select actions that don't exist or rarely appear in the dataset during training, and the true value of these actions cannot be estimated from offline data, easily leading to value overestimation. In practice, conservative penalties or constraints on policy deviation from the data distribution are typically used to mitigate this. For code agents that are trained on a fixed task set and then released, offline RL means the model's capability is "frozen" within the scope covered by the task set after release, unable to automatically adapt to new failure modes that emerge online.

Design Approach for Continuous Online RL

The other route treats the requests the agent is already serving as the task stream itself. The logic is straightforward: a request comes in, the model responds, the system scores the response, and the scoring result is used to train an update—but this update must pass through a gate before entering the serving weights.

According to the open-source implementation mentioned in the post, reef's OpenClaw^Ad-RL recipe is a concrete example of this route. As described on the recipe page: 72 GSM8K problems as the task stream, Qwen3-4B serving as both the policy model and the process reward model, Qwen3-32B as the student model, seven GPUs, and a threshold that must be met before an update is accepted. The learning curve is recorded session by session on the recipe page.

This design has two key differences from the offline loop, both related to serving rather than training:

Hot-swapping updates. Accepted updates are hot-swapped into the serving engine, with inference continuing to respond throughout the update process. The same tech stack serves and trains a LoRA adapter simultaneously. This means no downtime window and no dual-track operation.

Rejected updates never enter serving; accepted updates can be rolled back. Updates that fail the threshold never reach the serving weights. Accepted updates that later reveal problems can be rolled back by re-releasing a previous version.

These two mechanisms turn the boundary between "training" and "serving" from a wall into a valve.

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique that attaches two low-rank matrices alongside the weights of a pre-trained model, training only these two small matrices while freezing the original weights. Since the trainable parameter count is typically only a few percent of the original model, LoRA adapter files are extremely small and fast to load and switch, which makes "hot-swapping" an engineering-feasible operation: the serving engine can unload an old adapter and mount a new one without downtime. In continuous online RL, each gated update is often a new LoRA adapter, so training and serving can be deployed on the same tech stack (same base model plus different adapters), and rollback can be achieved in seconds by reloading the previous adapter version. This architecture of "base model stays fixed, adapters are replaceable" is the key engineering prerequisite for downgrading the wall between training and serving into a valve.

The Gating Mechanism Is the Real Core

The real divergence between the two routes isn't where the training data comes from, but what stands guard in front of the weights.

The gate in offline RL is the release process itself: human review, offline evaluation, canary deployment. It's slow, but every step is traceable. The gate in continuous online RL is an automatic threshold: an update must meet a certain standard on the scoring stream to be accepted. It's fast, but the reasonableness of the threshold itself requires continuous validation.

The post author also pointed out the limitations of the current implementation. GSM8K makes scoring completely unambiguous—the answer is either right or wrong—which makes the entire recipe exceptionally clean. But in real scenarios, most tasks don't have such a ready-made verifier. So he argues that the valuable next step is to find a task with a built-in verifier to test this pipeline, rather than staying on benchmarks like GSM8K that have been validated countless times.

Another limitation is that a single learning curve is the result of one run, with all conditions documented on the page, but it's not a benchmark. A single run can't prove that this method works under different conditions.

GSM8K (Grade School Math 8K) is a benchmark dataset containing about 8,500 grade-school math word problems, each with a clear numerical answer, so exact-match automatic scoring can be used. It's widely used in RL and reasoning model research because the reward signal is "unambiguous"—the model outputs a final answer, and a program can compare it to give a 0 or 1 reward, without needing human annotation or an additional reward model to judge correctness. This built-in-verifier property makes the training pipeline exceptionally clean, but it also masks the difficulty of real scenarios: most real tasks (such as coding, writing, customer service) don't have ready-made programmatic scorers, and rewards need to be learned or human-evaluated, which is noisy and prone to gaming. Therefore, the article emphasizes that running a continuous RL pipeline on GSM8K only proves the mechanism is feasible, not that it applies to real business tasks with ambiguous reward signals.

What It Means for People Doing Agentic RL

If you're working on agentic RL, these two questions are worth answering seriously:

Does your training data come from a fixed set or from online serving traffic? If the former, can your retraining cycle keep up with the speed of distribution drift? If the latter, how is your gating threshold set, what's your rejection rate, and has your rollback mechanism been tested?

A more fundamental question: when the agent itself is a continuously running service, where should the boundary between training and serving be drawn? Offline RL's answer is to separate them; continuous online RL's answer is to connect them through a gate. Both answers have merit, depending on how you weigh auditability versus adaptability.

From an engineering perspective, the appeal of continuous online RL is that it turns "model release" from an endpoint into a starting point. But the cost of this shift is introducing new failure modes: gates being bypassed, threshold drift, unclear rollback trigger conditions. These problems simply don't exist in offline RL, because in offline RL the model and serving are decoupled.

Which route to choose ultimately depends on what you fear more: the model failing to keep up with the real distribution, or automatic updates bringing uncontrollable risk.

One-line Summary

After a model ships, should training data come from a fixed task set or online serving traffic? The key divergence is in the gating mechanism.

Paragraph Summary

Starting from the case of Microsoft's 4B coding agent, the article discusses the training challenge after a reinforcement learning agent is released. Offline RL relies on a fixed task set and release process, with clear scoring and auditability, but once deployed, the task set can't represent the real distribution, and failure modes exceed what designers imagined. The other route is continuous online RL: treating the requests the agent is already serving as the task stream, with the system scoring responses and training updates, but updates must pass a gating threshold before entering serving weights. reef's OpenClaw-RL recipe (72 GSM8K problems, Qwen3-4B as policy and reward model, Qwen3-32B as student, seven GPUs) is a sample of this route, supporting hot-swapping and rollback of updates. The author notes that the real divergence between the two routes isn't the data source, but what stands guard in front of the weights: offline relies on a manual release process, online relies on automatic thresholds. The current limitation is that GSM8K's built-in verifier makes scoring too clean, and a single learning curve doesn't constitute a benchmark.