Qwen Open-Sources QwenGyre: An Elastic Reinforcement Learning Framework for Extremely Long-Horizon Agents

Training AI agents capable of completing extremely long, complex tasks has always been one of the thorniest problems in reinforcement learning engineering. The Qwen team's newly open-sourced QwenGyre attempts to tackle this head-on — it brings elastic scheduling and trajectory processing to extreme-long-horizon (XLong) agent training, and validated the approach on a model at the 2.4T parameter scale.

Why Long-Horizon Agent Training Is So Hard

Traditional agent RL training mostly assumes tasks are completed within a few hundred to a few thousand tokens. Once tasks become real software engineering tasks requiring multiple rounds of tool calls, sub-agent collaboration, and failure retries, a single rollout easily exceeds a hundred thousand or even several hundred thousand tokens, and the compute and engineering complexity of the entire training loop spirals out of control.

QwenGyre is positioned precisely for this scenario: extreme-long-horizon (XLong) agent training. What it must handle isn't just long sequences, but also the constantly forking behavior during execution — context compression, spawning sub-agents, failure retries — all of which cause the number of training samples to explode, with a large portion of the content being highly redundant.

What is a rollout in RL training?

In the context of reinforcement learning, a rollout refers to the process of letting the current policy model actually execute a task in an environment and collect trajectory data. For a language model agent, one rollout begins with receiving a prompt, and the model generates tokens step by step, calls tools, observes the returned results, and continues making decisions until the task terminates or the maximum number of turns is reached — the entire record becomes one trajectory sample. During the training phase, these fixed trajectories are used to compute policy gradients and update parameters.

The key point is that the resource characteristics of the two are completely different: rollout is autoregressive, serial, with memory usage growing continuously as the context expands, and highly unpredictable latency; training is batched, parallel, with stable compute density. In traditional short-task settings, the two can be executed in turns on the same batch of GPUs without conflict; but when the horizon stretches to hundreds of thousands of tokens and trajectory lengths vary enormously, who finishes first and who is waiting becomes a sharp scheduling problem. This is precisely the structural contradiction that Elastic Scheduler aims to decouple.

Two Core Designs: Elastic Scheduler and Trajectory Processor

Elastic Scheduler: Letting GPUs Flow Elastically Between Rollout and Training

A structural contradiction in agent training is that rollout (the model actually executing tasks) and training (updating parameters) have compute demands that fluctuate drastically over time. Traditional approaches either place both on the same batch of GPUs (Colocate) or decouple them with an asynchronous pipeline — the former has them competing for resources, the latter introduces policy lag.

QwenGyre's Elastic Scheduler dynamically reallocates GPUs between rollout and training. Crucially, this reallocation is "live" — tasks currently in execution are not interrupted, avoiding trajectory invalidation caused by resource switching. Official data shows up to a 1.85× speedup compared to the Colocate approach and 1.78× compared to the Async approach.

Colocate and Async: Two Classic Scheduling Paradigms

Colocate means placing rollout and training on the same batch of GPUs and executing them in turns: part of the time is spent on sampling, part on updates, sharing the resource pool with low communication overhead, but the two compete for memory and compute, and must be strictly serialized — any slower phase drags down overall throughput. Async splits the two onto different GPU groups running in parallel, so sampling and training don't block each other, yielding higher throughput, at the cost of "policy lag" — the data used for training comes from a slightly earlier version of the model, creating a distribution shift from the current policy, which may harm convergence stability and final performance.

QwenGyre's Elastic Scheduler attempts to capture the advantages of both: the GPU pool no longer has fixed ownership, but is dynamically allocated based on the real-time pressure of the rollout queue and training queue, and the migration process doesn't interrupt tasks in execution, avoiding the waste of "discarding half a trajectory just to reallocate resources." The 1.85× (vs. Colocate) and 1.78× (vs. Async) speedups essentially come from this compromise that "neither competes for resources nor sacrifices policy freshness."

Trajectory Processor: Organizing Forked Execution into Prefix-Shared Trees

If the scheduler solves "how to divide compute," the Trajectory Processor solves "how to manage samples."

A single agent execution is naturally a tree rather than a line: the same prefix may branch into continuing after compression, dispatching a sub-agent, or directly retrying, among multiple branches. If stored leaf by leaf, samples would be massively redundant. QwenGyre organizes these branches into prefix-shared trajectory trees, yielding a bounded, deduplicated set of training samples.

An easily overlooked but very practical design: even if a task only partially completes after a timeout, that partial progress is still scored. This means that "unfinished branches" on a tree can also contribute gradient signals rather than being discarded outright — in long tasks, this can significantly improve sample utilization.

Why Prefix-Shared Trees Save Samples

Suppose an agent task decides at some step to spawn a sub-agent, and then falls back to the same position to retry after a failure — the first N tokens of these two branches are completely identical. If stored leaf by leaf, the same prefix would be saved redundantly many times; in a long task that repeatedly compresses context and repeatedly retries, this redundancy could reach several-fold or even orders of magnitude.

The prefix-shared trajectory tree approach merges common prefixes into a single node, splitting branches only at true forks, with the entire tree sharing storage and computation. This brings two direct benefits: first, the training sample set becomes "bounded," no longer expanding infinitely due to the recursive nesting of sub-agent spawning; second, the KV cache and attention computation of the forward pass can be reused on shared prefixes, significantly reducing memory and compute overhead under long contexts. Combined with the design that "partial progress after timeout still counts toward scoring," compressed, interrupted, and misrouted branches are no longer pure waste but become effective sources of gradients.

Results: Performance on Qwen3.8-2.4T-A95B

QwenGyre was validated on a large-scale MoE model, Qwen3.8-2.4T-A95B, with a single rollout on the order of about 700,000 tokens.

On NL2RepoBench (a natural language to code repository generation task), QwenGyre improved the score from 52.5% to 58.5% in just 48 steps, with end-to-end training time reduced by about 44% compared to the Async approach. This combination is worth noting: it both improves the score and is fast — usually improvements in long-horizon RL come at the cost of exploding training time.

Paper link: arxiv.org/abs/2609.33848

What MoE and the A95B Naming Mean

The 2.4T in Qwen3.8-2.4T-A95B refers to the model's total parameter count of approximately 2.4 trillion, and the A in A95B stands for Activated, meaning the number of parameters actually activated per forward pass is about 95 billion. This structure of "large total parameters, small activated parameters" is the Mixture-of-Experts model (MoE): the feed-forward layers are split into multiple experts, and each token is routed to only a few experts for computation, so inference cost roughly scales with the activated parameter count rather than the total parameter count.

MoE is especially friendly to long-horizon agent training — it makes sequences on the order of 700,000 tokens in a single rollout computationally feasible, while maintaining far greater capacity than dense models to remember complex tool-calling patterns and failure-recovery strategies. Understanding this is essential to appreciating why QwenGyre's scheduling and trajectory optimizations are worth validating on a model of this scale: the larger the parameter count and the longer the sequences, the more severely scheduling waste and sample redundancy at the training framework level are amplified, and the more considerable the optimization gains.

Implications for Agent Training Engineering

The value of QwenGyre lies not in any single algorithm, but in how it rewrites, one by one, several engineering assumptions that had been tacitly accepted in long-horizon agent RL: compute allocation can be elastic without interrupting execution, forked trajectories can be structurally reused, and partial progress can count as training signal.

For teams working on agent post-training or RL infrastructure, this design philosophy is more worth referencing than the specific speedup numbers — it suggests a direction: as task horizons grow longer, the focus of the training framework shifts from "how to efficiently run one step" to "how to efficiently manage an execution tree that keeps forking, keeps timing out, and keeps repeating."

One-line Summary

Qwen open-sources QwenGyre, using elastic scheduling and prefix-shared trajectory trees to crack the challenge of extreme-long-horizon agent training.

Paragraph Summary

QwenGyre is a training framework open-sourced by the Qwen team for extreme-long-horizon (XLong) agent reinforcement learning, directly targeting the pain points of real software engineering tasks where a single rollout often runs to hundreds of thousands of tokens and constant forking during execution causes sample explosion. It has two core designs: Elastic Scheduler dynamically and non-disruptively reallocates GPUs between rollout and training, achieving up to 1.85× speedup vs. Colocate and 1.78× vs. Async; Trajectory Processor organizes forks such as context compression, sub-agent spawning, and failure retries into prefix-shared trajectory trees, yielding a bounded, deduplicated sample set, and allows partial progress after timeout to still contribute gradients. On the MoE model Qwen3.8-2.4T-A95B with 2.4T total parameters and 95B activated, a single rollout runs to about 700,000 tokens, and on NL2RepoBench it rose from 52.5% to 58.5% in just 48 steps, with end-to-end training time reduced by about 44% vs. Async.