JHiMA-WebRL: Hierarchical Multi-Agent RL Breaks Through Long-Horizon Web Tasks

Multimodal web agents are gradually learning to operate browsers like humans, but real-world web tasks often require long-horizon planning across pages and multiple steps, where a single policy model can easily lose its way. A recent study from arXiv proposes the JHiMA-WebRL framework, which attempts a joint training approach of "planner + executor" to enable 7B-level small models to match or even surpass larger monolithic models on long-horizon web navigation tasks.

Long-Horizon Web Tasks: Why Monolithic Policies Struggle

Most existing web agents adopt an end-to-end monolithic policy that directly outputs the next action based on the current screenshot and interaction history. This approach performs reasonably well on short tasks, but when a task requires crossing multiple pages and dozens of interaction steps, the model must simultaneously handle high-level planning, state tracking, and visual grounding—threefold burden that easily breaks the reasoning chain.

Previous work has attempted to introduce independent planners to decompose tasks, but a key flaw is often overlooked: the planner typically does not participate in subsequent joint updates and only generates subtasks once, so deviations during execution cannot feed back to the planning layer. The researchers thus raise a core question—if planning and execution are explicitly separated and jointly trained in reinforcement learning, can this form a stronger inductive bias?

Core Design of JHiMA-WebRL: Decomposer and Executor in Collaboration

JHiMA-WebRL splits the web agent into two roles: a task decomposer and a subtask-conditioned actor. The decomposer infers high-level subtasks from multimodal GUI trajectories, while the actor performs visual grounding and concrete operations given the subtask condition.

Training proceeds in two stages. In the first stage, a strong teacher model constructs structured subtask supervision signals from multimodal GUI trajectories, and these annotations are used to fine-tune both the decomposer and the executor simultaneously, establishing an initial foundation for collaboration. In the second stage, Group Relative Policy Optimization (GRPO) is extended to the dual-role setting, jointly optimizing both modules in the interactive WebArena environment through hierarchical rollout grouping and role-specific advantage estimation.

The cleverness of this design lies in the fact that the decomposer is no longer a static "one-shot planner" but can continuously adjust its policy based on execution feedback, forming a closed loop between planning and execution.

GRPO (Group Relative Policy Optimization) is a reinforcement learning algorithm that requires no value network, originally used for reasoning alignment of large language models. Its core idea is to sample a group of outputs for the same input and normalize each output's advantage using the mean and standard deviation of rewards within the group, thereby avoiding training an additional critic model and significantly reducing memory and computation overhead. In standard GRPO, all outputs share the same policy; JHiMA-WebRL extends it to a dual-role setting, requiring separate reward signals and role-specific advantage estimation for the decomposer and executor, while hierarchical rollout ensures that samples within the same group come from the same subtask context, making intra-group comparisons meaningful. This modification is the key technical detail enabling stable convergence of joint training.

Empirical Performance: 7B+7B Combination Outperforms 32B Monolithic Model

On the WebArena-Lite-v2 benchmark, JHiMA-WebRL-14B, composed of a 7B executor and a 7B decomposer, achieves an average success rate of 37.2%. For comparison, the ScaleCUA-32B agent fine-tuned with reinforcement learning on the same training tasks achieves a success rate of 24% under the same evaluation protocol.

The parameter comparison is noteworthy: the 14B combined model significantly leads the 32B monolithic model in success rate, indicating that the structural advantages brought by role decomposition can partially substitute for parameter scaling. Ablation experiments further validate the value of joint updates—updating the decomposer during the RL stage yields better results than updating only the executor.

WebArena-Lite-v2 is a lightweight revised version of the WebArena benchmark. WebArena constructs multiple self-hosted replicas of real websites (e.g., e-commerce, forums, code hosting platforms) and requires agents to complete cross-site long-horizon tasks through a browser, with success judged by functional correctness rather than text matching. The Lite version streamlines the task set and number of websites, reducing environment setup and execution costs while retaining the original benchmark's core examination of multi-step navigation and state tracking. v2 further corrects the executability and judgment logic of some tasks. Therefore, success rates on this benchmark can more realistically reflect an agent's long-horizon planning ability in interactive web environments, rather than simple static question-answering accuracy.

Implications for Web Agent Research

The significance of this work goes beyond a single benchmark number. It suggests a technical path different from "making models bigger": through explicit role decomposition and joint reinforcement learning, small models can also achieve competitiveness on complex long-horizon tasks. For practical deployment, 7B-level models have a clear advantage in inference cost, which is particularly critical for product scenarios requiring high-frequency calls to web agents.

Of course, a 37.2% success rate also shows that long-horizon web tasks are far from solved, with most tasks still failing. Credit assignment between the decomposer and executor, the quality ceiling of teacher model annotations, and the training overhead brought by hierarchical rollout are all questions that need to be answered. However, the direction of joint learning of role decomposition offers an option worth following for the architectural design of web agents.

Credit assignment in multi-agent or hierarchical reinforcement learning refers to: when the final task succeeds or fails, how to reasonably attribute the global reward to individual sub-decisions or roles. In JHiMA-WebRL, the decomposer decides "what to do" and the executor decides "how to do it"; if the task fails, it is difficult to determine whether the subtask decomposition was wrong or the execution action was mistaken. If only the final success rate is used as a sparse reward, the decomposer may receive misleading signals; if intermediate rewards are designed for the decomposer, they must rely on teacher annotations or heuristic rules. The quality of credit assignment directly affects whether joint training can enable the two modules to collaborate effectively, rather than one free-riding or passing the buck—this is also an important focus for future improvements of the framework.

One-line Summary

JHiMA-WebRL uses joint reinforcement learning of a planner and executor to enable a 7B+7B combination to surpass a 32B monolithic model on long-horizon web tasks.

Paragraph Summary

The JHiMA-WebRL framework addresses the problem of excessive planning and execution burden on monolithic policy models in long-horizon web navigation tasks by splitting the agent into a task decomposer and a subtask-conditioned executor. Training proceeds in two stages: first, a strong teacher model constructs subtask supervision signals for fine-tuning; then GRPO is extended to a dual-role setting, jointly optimizing both modules through hierarchical rollout and role-specific advantage estimation, enabling the planner to continuously adjust based on execution feedback. On WebArena-Lite-v2, the combination of a 7B executor and a 7B decomposer achieves a success rate of 37.2%, significantly higher than the 32B monolithic model fine-tuned under the same protocol (24%). Ablation experiments confirm that jointly updating the decomposer outperforms updating only the executor, indicating that explicit role decomposition can partially substitute for parameter scaling and providing a new path for small models to handle complex long-horizon tasks.