A New Way to Define GUI Agent Tasks: From Action Sequences to Target Screen States

A Small Shift with Far-Reaching Implications

In GUI (Graphical User Interface) agent development, there has long been a default modeling paradigm: defining a task as a series of concrete operation steps — clicking a button, entering some text, dragging a slider. This "action sequence" approach is intuitive and easy to understand, yet it has revealed increasing fragility in real-world applications.

Recently, a developer on Reddit raised a thought-provoking idea: instead of defining a GUI task as a chain of clicks, define it as "the target state the screen should display after the task is complete." This shift may sound trivial, but it could fundamentally change how we evaluate and build GUI agents.

"I've started thinking of GUI tasks as a target screen state rather than a series of clicks. The agent can take any valid path, but the task itself defines what the UI should look like when it's done."

Why the Action Sequence Paradigm Is So Fragile

The Problem of Path Dependency

Traditional action sequence definitions have a core flaw: they assume there is only one correct path to complete a task. In real software interfaces, however, the same goal can typically be achieved in multiple ways.

Take "save a file" as an example. A user can click the save icon in the toolbar, press the keyboard shortcut Ctrl+S, or navigate through the menu bar via "File → Save." If we hardcode the task as "click the third icon in the toolbar," the evaluation system will incorrectly mark the task as failed whenever the UI layout changes slightly or the agent chooses an equivalent alternative path.

Path dependency is especially pronounced in reinforcement learning and imitation learning frameworks. Many current GUI agents are trained via Behavior Cloning, where the model learns the distribution of actions from human demonstration trajectories rather than the essential goal of the task. This makes models extremely sensitive to distribution shift — even when the task objective is identical, a slight difference between the runtime environment and training screenshots can cause the action sequence to collapse. This fragility is directly tied to path dependency: the model has memorized "click at this pixel location" rather than "find and activate the save function."

Interface Iteration Renders Evaluations Obsolete

Software interfaces continuously evolve with version updates — button positions, menu structures, and widget styles can all change. Task definitions based on action sequences are tightly coupled to specific UI layouts, meaning every interface update can invalidate existing benchmarks and require re-annotation and maintenance. This fragility significantly increases the engineering cost of GUI agent research.

Core Advantages of Target State Definition

Simpler and More Reliable Validation Logic

Once a task is defined as a "target screen state," the evaluation logic becomes clear: simply check whether the final interface has reached the expected state, regardless of the path the agent took. As the original post notes, "this makes success much easier to verify, especially when there are multiple ways to reach the same result."

This outcome-oriented validation approach naturally accommodates diverse agent behavior. Whether the task is completed via keyboard shortcuts, mouse clicks, or menu navigation, success is declared as long as the final screen state matches expectations. This aligns closely with how humans intuitively judge task completion — we assess whether something is done by looking at the result, not the process.

Significantly Improved Agent Robustness

The target state paradigm also has a deeper value: it encourages agents to autonomously explore effective paths rather than rigidly reproducing a preset sequence. When one path is blocked (e.g., a button is disabled), the agent is motivated to seek alternatives, because its goal is to reach a state rather than execute fixed actions. This flexibility is precisely what current GUI agents lack most, and it is a key reason they frequently fail in real-world environments.

Implications for Benchmark Design

Limitations of Existing GUI Evaluation Benchmarks

Many mainstream GUI agent evaluation benchmarks still score based on "action matching" or "step reproduction." This evaluation method has systematic biases: it may penalize agents that take reasonable but unexpected paths, while failing to truly measure whether the task was effectively completed.

The original poster explicitly expressed hope: "I'd love to see more benchmarks define tasks this way." This is essentially a call for the community to re-examine the methodological foundations of GUI evaluation.

Implementation Challenges of State Definition

Of course, defining GUI tasks as target states is not without difficulty. The central challenge lies in how to precisely and verifiably describe a "screen state." This may involve several dimensions:

• Visual-level state assertions: Determining whether the interface matches expectations via screenshot comparison or visual feature recognition;
• Semantic-level state descriptions: Extracting and validating key attributes from the UI's accessibility tree or DOM structure;
• Data-level verification: Checking whether the task truly changed the underlying data (e.g., the file was actually saved, the record was actually created).

How to strike the right balance across these three dimensions — ensuring robust validation without over-relying on any specific representation — is a key design challenge for this type of benchmark.

A Deeper Meaning: From Imitation to Goal-Directed Intelligence

This conceptual shift reflects a fundamental evolution in agent design philosophy. The action sequence paradigm is essentially an "imitation learning" mindset — having the agent reproduce human operation trajectories. The target state paradigm, by contrast, is closer to "goal-directed" intelligence — given a desired outcome, let the agent autonomously plan how to achieve it.

In the long run, truly practical GUI agents must be goal-directed. When humans use software, what's in their mind is "I want to export this report as a PDF," not "I need to click here, then click there." Letting agents anchor their understanding of tasks in outcomes — rather than mechanically memorizing operation paths — is what enables them to truly grasp the essence of a task.

The goal-directed paradigm aligns closely with classical frameworks in AI planning. In formal planning systems like PDDL (Planning Domain Definition Language), tasks are always defined as transitions from an initial state to a goal state, not as fixed action sequences — the planning algorithm is responsible for autonomously searching for paths that satisfy the goal. Introducing this idea into GUI agents effectively frames GUI interaction as "state-space search" rather than "sequence prediction." This shift also has downstream implications for how agents are trained: reinforcement learning based on outcome rewards (such as RLHF or sparse rewards based on task completion state) will better cultivate agents with genuine goal-understanding capabilities than pure imitation learning.

Conclusion

This observation from the Reddit community — just a brief shared thought — touches on a fundamental modeling question in the GUI agent field. Redefining tasks from "how to do it" to "what it should look like when done" may seem like a minor adjustment in evaluation perspective, but it could significantly improve both agent robustness and evaluation validity.

As GUI agents gradually move from the lab into real-world application scenarios, building evaluation systems that are both flexible and reliable will become a decisive factor in whether this technology can be successfully deployed. The target state paradigm may be precisely the key to opening that door.

Background: Accessibility Tree

The Accessibility Tree is a structured interface representation provided by operating systems, designed for assistive technologies such as screen readers. It abstracts each GUI control into a node with attributes like role, name, and state, forming a hierarchical tree structure. Compared to raw pixel screenshots, the accessibility tree is naturally robust to visual style changes — even if a button changes color or position, the validation logic remains effective as long as its semantic attributes are unchanged. Leading GUI agent benchmarks (such as OSWorld and WindowsAgentArena) have begun incorporating the accessibility tree for state verification, but how to define a "sufficiently comprehensive" semantic snapshot remains an open question.