IGG Benchmark: A New Challenge in Interactive GUI Grounding — How Can AI Agents Break Through Visibility Constraints?

Introduction: When a GUI Element "Disappears," Can AI Still Click It?

Current mainstream GUI grounding benchmarks typically assume that the target element is already displayed in a static screenshot, and the agent only needs to locate and click it. But real interfaces often do not satisfy this premise: buttons may be off-screen, appear only on hover, be obscured by pop-ups, or require waiting for loading before they become operable. This ability to "first make the target visible, then click" is precisely the blind spot of existing evaluations.

To address this, researchers proposed the Interactive GUI Grounding (IGG) benchmark, which requires agents to first make the target visible through interactive actions (scroll, move, drag, wait, click), and then complete the click. IGG contains 601 tasks, covering 16 controllable GUI environments and 7 subtypes of visibility constraints. Human participants achieved 94.6% accuracy in the test, while GPT-5.5 reached only 79.4%, and the strongest open-source vision-language model (VLM) only 31.6%. This significant gap reveals the core shortcoming of current AI agents in real interactive scenarios.

GUI grounding is a foundational task in the GUI agent field: given a natural language instruction, the agent needs to locate the corresponding interactive element (such as a button, input box, or link) in the interface screenshot and output its coordinates to perform a click. Early work was mostly based on datasets such as Rico and ScreenSpot, adopting the "static screenshot + element annotation" paradigm, where the model essentially performs target detection under vision-language alignment.

Vision-language models (VLMs) are multimodal large models that process both image and text inputs, such as GPT-4o and Qwen2.5-VL. In GUI scenarios, VLMs are often used as end-to-end agents: input the current screenshot and task description, and directly output the next action. Unlike traditional modular pipelines (perception → planning → execution separated), VLM agents need to simultaneously perform interface understanding, state inference, and action decision-making within a single model, which also makes "interaction planning" an area where their capability shortcomings are more easily exposed.

Performance Gap: Humans 94.6% vs. Open-Source Models 31.6%

In the human study, participants achieved an accuracy of 94.6%, indicating that these tasks are not difficult for humans. But AI models show polarized performance: GPT-5.5 reached 79.4%, performing relatively well; while the strongest open-source VLM (such as Qwen3-VL) achieved only 31.6% accuracy. Why is the gap so large?

Researchers pinpointed the root cause through two controlled interventions.

Knowing "How to Interact" ≠ Being Able to "Execute Interaction"

First intervention: directly telling Qwen3-VL which interactive action to perform (e.g., "scroll down three notches"), its accuracy only increased from 30.8% to 33.9%, almost no substantive improvement. Second intervention: the system executes the interactive actions on behalf of the model, and the model is only responsible for clicking after the target is exposed; accuracy jumped to 81.9%.

This comparison clearly shows: the bottleneck lies in executing the reveal interaction, not in recognizing the target after it is exposed. The model does not lack understanding of the target's appearance; rather, it struggles to plan and execute a series of actions to change the interface state so that the target enters the clickable range.

Controlled intervention is an experimental design commonly used in cognitive science and AI evaluation: while keeping other conditions unchanged, artificially change one variable (here, "whether to inform the action" or "whether to execute the interaction on behalf of the model"), observe the difference in model performance, and thereby decompose overall performance into specific sub-capabilities.

The two interventions in this article form a pair of "upper bound–lower bound" comparisons: informing the action is equivalent to giving the model a strong prior, testing its execution ability; executing the interaction on behalf of the model directly skips the reveal stage, testing its pure grounding ability. The huge gap between the two (33.9% vs. 81.9%) indicates that failures mainly occur in the "state change" step, not the "target recognition" step. The value of this experimental paradigm lies in refining the vague "the model is not good enough" into attributable capability gaps, providing clear targets for the construction of subsequent training data.

Capability Is Learnable: Improvements Across Model Families

The study further verified that this "visibility restoration" capability can be acquired through training. After fine-tuning on multiple model families, agents' performance on IGG consistently improved, indicating that this capability is not an accidental characteristic of a particular model, but a transferable skill.

Design Details of the IGG Benchmark

The IGG benchmark contains 601 tasks, covering 16 controllable GUI environments, each simulating different interaction logic and interface layouts. Tasks are divided into 7 subtypes of visibility constraints, for example:

• Off-screen targets: require scrolling or dragging to come into view
• Hover-triggered: targets appear only when the mouse hovers
• Occluded targets: covered by other elements, requiring closing or moving the obstruction first
• Delayed appearance: require waiting for loading or animation to complete

The action space is compressed into five basic operations: scroll, move, drag, wait, and click. This compact design reduces the complexity of the action space while retaining the core elements of real interaction.

Action space refers to the set of operations an agent can perform in an environment. Real GUI operations could include arbitrary coordinate clicks, multi-finger gestures, keyboard input, and other continuous or high-dimensional actions, but in evaluation they are usually discretized into a few atomic actions. IGG compresses the action space into five types: scroll, move, drag, wait, click. On one hand, this reduces the search space for policy learning, making comparisons between different models fairer; on the other hand, it retains the minimal operation set required to "change the interface state."

It is important to distinguish between atomic actions and action sequences: the former is a single-step execution unit, while the latter is a multi-step trajectory planned by the agent to achieve a goal. The core difficulty of IGG lies precisely in the latter—the model must infer the complete chain of "current target invisible → what action to perform → how many times to execute the action → whether the target is exposed → then click," rather than merely predicting a single-step action.

Why Is This Benchmark Worth Attention?

GUI agents are moving from "static screenshot understanding" to "dynamic environment interaction." The IGG benchmark fills the evaluation gap under visibility constraints, and its value is reflected in three aspects:

1. Exposing the real gap: Existing benchmarks may overestimate the actual capabilities of agents. When the target is not in the initial field of view, open-source models' performance drops precipitously, which is a warning for Agents deployed in real software.
2. Pointing out the direction for improvement: The bottleneck is not in perception but in execution, meaning future research should focus more on learning interaction policies rather than merely improving visual recognition accuracy.
3. Providing trainable signals: IGG tasks can be used for fine-tuning to help models learn the reasoning chain of "first change the state, then locate the target."

For AI programming and AI application developers, this means that when building GUI automation tools, more attention should be paid to the agent's interaction planning ability, not just element recognition accuracy.

Fine-tuning refers to continuing to train a pretrained model with task-specific data to adapt it to downstream capabilities. The significance of "visibility restoration capability can be acquired through training" described in this article lies in distinguishing two sources of capability: first, zero-shot generalization (the model can complete the task without training on it), and second, learnability (the model can master it after receiving relevant supervision signals). Experiments on IGG show that the 31.6% of open-source VLMs is closer to "lack of training signals" rather than "capability ceiling," which provides a clear path for subsequent work—constructing interaction trajectory data and performing supervised fine-tuning or reinforcement learning can significantly improve performance.

For developers, this also means that when selecting models, one should not only look at scores on static grounding leaderboards, but also pay attention to whether the model has been trained on interactive trajectories and its robustness in dynamic scenarios such as scrolling and hovering.

Conclusion: From "Seeing" to "Making Visible"

The IGG benchmark advances GUI grounding evaluation from static perception to dynamic interaction, revealing the core bottleneck of current AI agents under visibility constraints. What humans do almost effortlessly—"scroll a bit and then click"—is a chasm that models need to cross. The good news is that this capability can be acquired through learning. As more researchers focus on interactive grounding, we have reason to expect agents to become more reliable and autonomous in real software environments.

One-line Summary

IGG benchmark requires agents to interactively make GUI targets visible before clicking, revealing that open-source VLMs achieve only 31.6%, far below humans' 94.6%.

Paragraph Summary

Researchers proposed the Interactive GUI Grounding benchmark IGG, which requires agents to first make targets visible through actions such as scrolling, hovering, dragging, and waiting before completing a click. It contains 601 tasks, 16 environments, and 7 visibility constraint subtypes. Human accuracy reaches 94.6%, GPT-5.5 achieves 79.4%, and the strongest open-source VLM only 31.6%. Controlled experiments show that telling the model which action to perform only raises accuracy from 30.8% to 33.9%, while executing the interaction on behalf of the model jumps to 81.9%, proving that the bottleneck lies in executing the reveal interaction rather than recognizing the target. Fine-tuning experiments show that this capability can be learned across model families, pointing the direction for GUI agent research from static perception to interaction policies.