Long-Horizon Agentic RL: Don't Watch tokens/s, Watch the Trainer Idle

Why tokens/sec is misleading

In long-horizon agentic reinforcement learning (agentic RL), people habitually use tokens/sec to measure inference efficiency. But a recent analysis on Reddit points out that this metric is basically misleading in GRPO-style training. The reason is that all samples within a group must finish before training can begin: a task generates multiple sibling attempts that score each other, and the entire group only becomes trainable once all of them have completed.

As long as one attempt is stuck installing dependencies or repeatedly running tests, the whole group stalls. Request throughput may look high, but the trainer sits idle waiting. So the metric you should really track is: the time required to generate the next fresh, valid, trainable batch.

GRPO (Group Relative Policy Optimization) is a policy optimization algorithm that does not require a critic network. Its core idea is to sample a group of multiple candidate responses for the same prompt and estimate the advantage function using the relative ranking of rewards within the group (rather than absolute values). This design eliminates the value model, but the cost is that all samples within the group must fully finish generating before the mean and variance can be computed and the advantages normalized. In agentic RL, each candidate response is not a single forward pass but an entire multi-step tool-calling trajectory (installing dependencies, running tests, reading and writing files), and the time differences can be tens of times. As a result, the slowest trajectory in the group becomes the bottleneck for the entire batch. Understanding this makes it clear: tokens/sec measures the rate at which the model emits tokens, while what training actually waits for is the completion of the slowest trajectory. There is no causal relationship between the two.

Three engineering techniques that change batch collection efficiency

To address the problem of stragglers slowing down training, the original article offers three practical optimization paths, all centered on "shortening the wait."

Take groups in completion order, not start order

Don't wait for groups in start order; take groups in completion order. At the same time, keep a bounded surplus of in-flight candidate groups so that slow groups are bypassed rather than waited on. This is equivalent to introducing a pipeline at the scheduling layer, trading a small amount of extra concurrency for a trainer that always has work to do.

Straggler originates from distributed computing and refers to nodes or tasks in a cluster that complete significantly slower than their peers, often determining overall job time rather than the average. In GRPO-style group sampling, a straggler manifests as a trajectory that repeatedly retries, gets stuck on environment installation, or hits long-tail tests, while its sibling attempts in the same group are already ready. Bounded surplus is a classic fault-tolerant scheduling technique: the number of in-flight tasks is allowed to exceed what the target batch requires, but with a cap, and the excess is reclaimed in completion order, so slow groups keep running in the background without blocking the trainer. The cap prevents memory and sandbox resources from being exhausted indefinitely. Essentially, this trades a small amount of extra concurrency (surplus) for tolerance to long-tail latency—it is latency hiding, not latency elimination.

Split sandbox work into independent lanes

Split sandbox operations such as resets, repackaging commands, validation, and cleanup into independent lanes with pinned CPU sets, so that tool execution does not preempt serving resources. Tool execution and model serving competing for CPU is a common hidden bottleneck in long-horizon tasks.

Publish new weights one serving engine at a time

New weights are published one serving engine at a time, and unfinished trajectories are preserved rather than restarted. At the same time, an explicit upper bound K can be set on how stale a policy can be for returned groups. Under terminal-bench-style workloads (256 trajectories per batch), these changes reduce batch collection time from 591.8 seconds to 196.1 seconds.

In architectures where RL training and inference coexist, actor weights need to be periodically synced to serving engines (such as vLLM instances) to generate new trajectories. If weights are replaced all at once, in-progress trajectories must be interrupted and restarted, causing significant waste; rolling per-engine deployment allows unfinished trajectories to finish on the old weights. The sentinel tensors mentioned in the article are a placeholder/sentinel tensor mechanism: the training side first updates these sentinels, which then trigger or proxy the actual weight synchronization, thereby verifying whether the rollout pipeline is clear without copying the full actor weights. Therefore, this experiment validates the interface between weight updates and trajectory generation, not the full training loop including backpropagation and optimizer updates—a boundary that must be noted when interpreting its data.

Measured data and one key caveat

On SWE-bench, live weight updates account for about 11% of step time, with an average reward of 0.520 versus 0.527 with fixed weights—a very small gap. This shows that the overhead of live updates is manageable.

But the author explicitly cautions: live updates replace sentinel tensors, not the full actor weights. Therefore, what is validated is the rollout mechanism, not the complete training loop. The comparison between K=4 and K=8, along with the full step-by-step ablation breakdown, is in the original write-up.

Implications for teams doing agentic RL

First, monitoring metrics should shift from throughput to "trainer idle time" or "latency to the next trainable batch." Second, the scheduling strategy should allow bounded out-of-order processing, trading surplus for straggler tolerance. Third, resource isolation cannot stop at the GPU layer; CPU pinning and lane splitting also affect end-to-end efficiency.

Long-tail behavior in long-horizon tasks is inevitable; the engineering focus should be on keeping the long tail from blocking the main flow.

One-line Summary

In long-horizon agentic RL, tokens/sec masks trainer idle time; take groups in completion order and isolate resources.

Paragraph Summary

The article points out that in GRPO-style long-horizon agentic reinforcement learning, tokens/sec is a misleading metric: all sibling attempts within a group must finish before training can begin, and as long as one trajectory is stuck installing dependencies or running tests, the whole group stalls while throughput numbers still look good. What you should really track is the time required to generate the next fresh, valid, trainable batch. To this end, the original article offers three engineering techniques: take groups in completion order rather than start order, and maintain a bounded surplus to bypass stragglers; split sandbox resets, validation, and cleanup into independent lanes with pinned CPU sets to prevent tool execution from preempting serving resources; and roll out weights one serving engine at a time while preserving unfinished trajectories, with an upper bound K on policy staleness. Under terminal-bench-style workloads, batch collection time drops from 591.8 seconds to 196.1 seconds. On SWE-bench, live updates account for about 11% of step time, with reward 0.520 versus 0.527—manageable overhead, but the author cautions that what is replaced are sentinel tensors, not full actor weights, so what is validated is the rollout mechanism rather than the complete training loop.