Online Dense Reinforcement Learning + Planning Grounding: Enabling UI Navigation Agents to Learn While Operating on Real iOS Apps

Why do small models always "learn but never master" UI navigation?

Frontier vision-language models perform impressively on user interface (UI) navigation tasks, but their high inference costs make them difficult to use as everyday tools. On the other hand, small models trained via supervised fine-tuning (SFT) or offline reinforcement learning often merely memorize static action trajectories—once the interface changes slightly or an unexpected state appears, execution becomes brittle, let alone recovering from errors.

This contradiction points to a neglected link: the gap between the training environment and the real environment. Offline data cannot cover the dynamic feedback of real applications, while online learning faces dual constraints of cost and safety. A recent study proposes a targeted solution: train UI navigation agents directly on real iOS apps using online dense reinforcement learning.

The core idea is not to make models bigger, but to make the training paradigm "alive"—allowing the agent, through the division of planning and execution, to continuously obtain dense feedback from real interactions.

Online reinforcement learning vs. offline reinforcement learning are the two major paradigms of reinforcement learning categorized by data source. Offline reinforcement learning (offline RL) only uses a pre-collected fixed dataset for training; the agent cannot interact with the environment to generate new data, so it is easily limited by the coverage of the data distribution and performs brittle when encountering out-of-distribution states. Online reinforcement learning (online RL), in contrast, allows the agent to continuously interact with the environment during training, collecting data while updating the policy, and can explore states not covered by offline data—but at the cost of high interaction costs and potential safety risks in real systems. The method described in this article chooses the online paradigm precisely to use real interactions to compensate for the distribution blind spots of offline data.

Breaking down the framework: a dual-component design decoupling planning and execution

The key design of this work is to split the navigation agent into two specialized components, of which one explicitly identifiable module is "planning." This decoupling approach responds to the inherent structure of UI navigation tasks:

• The planning side is responsible for understanding interface semantics, decomposing task goals, and deciding what to do next;
• The execution side is responsible for translating planning intentions into concrete interface operation actions.

Separating the two types of capabilities has the benefit that: planning can focus on "where to go," and execution can focus on "how to click." When a certain link goes wrong, the decoupled structure also provides room for locating the problem and making local corrections, rather than letting an end-to-end model bear all the pressure in a black box.

The original abstract gives limited detail on the execution component, but what is certain is that the entire framework serves the same goal—completing closed-loop learning on a real-time running application, rather than relying on pre-recorded trajectory data.

Online dense reinforcement learning: "learning by doing" on real applications

"Online" and "dense" are the two pillars of this method.

Online means the agent directly interacts with a running iOS app and obtains real, immediate state feedback. Compared with offline data, this can expose more long-tail scenarios and also forces the model to face the uncertainty of real interfaces.

Dense refers to the design of the reward signal. UI navigation tasks inherently suffer from the sparse reward problem—there is only a clear success signal when the task is finally completed, and intermediate steps lack guidance. Dense reinforcement learning attempts to provide finer-grained feedback during the process to help the agent understand which intermediate decisions are valuable.

The direct benefit of this combination is alleviating the tendency of small models to "memorize trajectories." The model no longer only aligns with static samples, but adjusts its policy through continuous interaction, thereby improving its ability to recover from errors. The failure modes pointed out in the original text—brittle execution and inability to recover from errors—are exactly the problems this paradigm aims to solve.

Sparse reward vs. dense reward are two typical forms of reward design in reinforcement learning. Sparse rewards give success or failure signals only at the end of a task; for example, in UI navigation, returning 1 only when the final page is correct and 0 otherwise makes it difficult for the agent to judge which intermediate step is truly valuable among many steps, making the credit assignment problem prominent. Dense rewards provide fine-grained feedback at multiple intermediate states along the trajectory, such as giving partial rewards each time a subgoal is completed or each step closer to the goal, thereby providing richer gradient signals for policy optimization. The difficulty in designing dense rewards lies in how to define intermediate signals that are both accurate and automatically obtainable, avoiding the reward being exploited by "score farming."

The value of this work and the open questions

From the problem definition, this study hits the real bottleneck in deploying UI navigation agents: it is not that the model is not strong enough, but that the training method is disconnected from the deployment environment. Online dense reinforcement learning provides a direction that allows smaller models to maintain adaptability in dynamic environments.

However, the currently public information still leaves questions worth asking. The specific implementation of the planning component, how dense rewards are obtained without interfering with real applications, and how sample efficiency and stability of online training are guaranteed—these details determine whether the method can move from a framework to reproducible engineering practice.

For practitioners focused on AI applications and agents, the inspiration of this work may not lie in a specific module, but in the choice of training paradigm: when the ceiling of offline data becomes apparent, putting the learning process back into the real environment may be a pragmatic path for small models to catch up with frontier capabilities.

Sample efficiency refers to the number of interaction samples an agent needs to reach a certain performance level, and is one of the core metrics for deploying online reinforcement learning. Every interaction on a real iOS app comes with time and compute costs; if sample efficiency is low, the cost of online training may quickly spiral out of control. Common ideas for improving sample efficiency include: using a planning module to narrow the action space, reusing historical interaction experience, introducing prioritized sampling or model-assisted prediction, etc. In addition, the stability of online training also depends on reasonable exploration strategies and reward normalization; otherwise, the policy can easily oscillate or even degrade under the stochastic feedback of the real environment.