Visual Token Routing: Letting Computer-Use Agents Process Only Key Visual Information

Title

Visual Token Routing: Letting Computer-Use Agents Process Only Key Visual Information

Meta Description

Visual Token Routing (VTR) jointly trains a lightweight spatiotemporal router that lets computer-use agents keep only essential visual tokens. With just 2000 training steps, it matches full-resolution model performance on benchmarks like OSWorld, reduces visual tokens by 2.5-5.8x, cuts inference cost by 3.3x, speeds up prefill by 2.9x, and outperforms resolution-reduction and token-dropping baselines.

Article Content

Computer-use agents (CUAs) have already demonstrated superhuman performance on several long-horizon benchmarks, but their high computational cost limits their application in long-horizon tasks and real-time scenarios (such as video games). The core bottleneck is that each screenshot is represented as thousands of visual tokens, which accumulate as the agent reasons over long interaction histories, forming extremely long contexts. Research shows that most of these visual tokens are either semantically irrelevant to the agent's next action or temporally redundant across consecutive screenshots.

Based on this observation, researchers propose Visual Token Routing (VTR)—a joint training framework for learning sparse visual representations that retain only the information needed for action prediction. VTR jointly trains a vision-language model with a lightweight spatiotemporal router that inspects each image region in the current and previous screenshots and decides whether its visual tokens enter the language model. The router is optimized with an adaptive loss that minimizes the number of visual tokens while minimizing performance loss.

Core Problem: Redundancy and Irrelevance of Visual Tokens

The standard pipeline for CUAs is: screenshot → visual encoder extracts features → language model reasons about actions. The problem is that a single high-resolution screenshot may produce thousands of tokens, and the agent needs to review a large number of historical screenshots in long-horizon tasks. This causes context length to balloon rapidly, and inference cost to skyrocket.

Researchers observed two key phenomena: semantic irrelevance—many image regions (such as backgrounds and irrelevant UI elements) are unrelated to the next action; and temporal redundancy—consecutive screenshots change very little, and many tokens repeat along the temporal dimension. These two observations form the design basis of VTR: not all visual tokens are worth keeping, and the agent only needs to "see" the parts that actually affect decisions.

Visual tokens are the basic unit through which vision-language models process images. Images are typically divided into fixed-size patches (e.g., 14×14 or 16×16 pixels), and each patch is mapped by a visual encoder (such as ViT) to a feature vector, i.e., a token. A 1024×1024 high-definition screenshot can produce thousands of patches, corresponding to thousands of tokens. In the language model, these tokens are concatenated with text tokens and jointly participate in self-attention computation, whose cost grows quadratically with the number of tokens. Therefore, reducing the number of visual tokens is the most direct way to lower the cost of long-context inference.

How VTR Works: Spatiotemporal Router and Joint Training

The core of VTR is a lightweight spatiotemporal router. It simultaneously examines corresponding image regions in the current and previous screenshots to determine whether the visual tokens of that region should enter the language model. The router is not post-hoc pruning; it is jointly trained with the vision-language model, enabling the model to still make correct decisions under sparse visual input.

The training objective uses an adaptive loss: compress the number of visual tokens as much as possible while maintaining task performance. This means the router does not simply discard tokens but learns to retain the information that truly matters for action prediction. The entire training requires only 2000 steps to achieve performance comparable to full-resolution models on OSWorld, CUA-World, and video game action prediction tasks, while using 2.5 to 5.8 times fewer visual tokens.

This sparsification yields significant practical benefits: inference cost reduced by up to 3.3 times, and prefill speed improved by up to 2.9 times. For video game scenarios requiring real-time response, prefill acceleration means the agent can react faster—a key step toward real-time computer-use agents.

Prefill is the first stage of large language model inference: the model processes all tokens in the input sequence in parallel, computes and caches key-value (KV) pairs, and prepares for subsequent token-by-token generation. Prefill computation is proportional to input length and is the main source of latency in long-context scenarios. For computer-use agents that need to respond quickly, prefill speed directly determines the frequency of the agent's perception-action loop. VTR shortens the input sequence by reducing the number of visual tokens, thereby significantly accelerating prefill—which is precisely the key to its real-time application value.

Joint training means the router and the vision-language model are not optimized independently in separate stages, but updated end-to-end in the same training loop. The router decides token selection according to the current policy, while the language model learns action prediction based on the routed sparse input; the gradients of both are backpropagated together through the adaptive loss. This design avoids the disconnect between routing policy and downstream task in the "prune-then-fine-tune" paradigm, enabling the router to directly learn the region-selection policy that is most informative for action prediction.

Comparative Experiments: VTR Outperforms Resolution-Reduction and Token-Dropping Baselines

An intuitive alternative is to reduce input image resolution or use other token-dropping strategies. VTR was compared against these baselines in experiments, leading by up to 12.8 percentage points on OSWorld. This shows that intelligently selecting tokens is more effective than simple compression or uniform dropping—the router learns "which tokens matter for the current decision" rather than uniformly reducing information density.

This result also addresses a common question: why not simply use a smaller visual encoder? Because the visual information required varies dynamically across tasks and moments, and fixed compression loses details at critical moments. VTR's dynamic routing mechanism can adaptively retain information according to context, which is precisely its advantage.

Implications for the Computer-Use Agent Field

The value of VTR lies not only in reducing computational cost, but also in opening up two previously constrained scenarios: long-horizon tasks and real-time domains. In long-horizon tasks, agents need to process large numbers of historical screenshots, and token accumulation is the main bottleneck; real-time scenarios such as video games are extremely sensitive to inference latency. Through sparse visual representations, VTR makes agent deployment in these two scenarios more feasible.

The code and models are publicly available, and the researchers provide a project page (https://pranjal2041.github.io/vtr) for reproduction and further exploration. For researchers and engineers focused on AI agent efficiency, multimodal reasoning, and long-context processing, VTR offers a "learn to ignore" approach: rather than letting the model see everything, let it learn to see only what matters.

As computer-use agents move from benchmarks to real-world applications, this kind of efficiency optimization work will become increasingly critical. VTR demonstrates the potential of jointly trained routers and points the way for future research—sparsifying visual tokens is not a compromise that sacrifices performance, but a win-win that can be achieved through learning.

One-line Summary

VTR sparsifies visual tokens by jointly training a spatiotemporal router, significantly reducing the computational cost of computer-use agents.

Paragraph Summary

Computer-use agents perform excellently on long-horizon benchmarks, but each screenshot produces thousands of visual tokens, and long contexts lead to high computational cost. Research finds that many visual tokens are semantically irrelevant to actions or redundant across consecutive screenshots. To address this, researchers propose Visual Token Routing (VTR), which jointly trains a vision-language model with a lightweight spatiotemporal router to adaptively decide which image regions' tokens enter the language model, using an adaptive loss to compress tokens while maintaining performance. With only 2000 training steps, VTR matches full-resolution models on OSWorld, CUA-World, and video game action prediction, reduces visual tokens by 2.5 to 5.8 times, cuts inference cost by up to 3.3 times, improves prefill speed by up to 2.9 times, and leads resolution-reduction and token-dropping baselines by up to 12.8 percentage points on OSWorld.