GUI-JEPA: A Decoder-Free GUI Agent with 4× Speedup

Mobile GUI agents typically rely on vision-language models (VLMs) with 7 billion parameters, deciding clicks or swipes by generating text token by token. While flexible, this paradigm brings two prominent problems: inference speed is limited by autoregressive decoding, and keeping large models resident in memory drives up deployment costs. A recent paper titled "GUI-JEPA: Candidate-Conditioned JEPA Latents as a Fast Action Policy for Mobile GUI Agents" proposes a different approach—bypassing text decoding entirely and performing action selection directly in latent space.

From "Generating Actions" to "Evaluating Candidates"

The core insight of GUI-JEPA is that the action space of GUI operations is inherently discrete and enumerable. Rather than having the model generate "click coordinates (320, 480)" character by character, it is better to pre-list a set of candidate actions and let the model quickly determine which candidate best matches the current instruction.

The research team built a policy network based on the Joint Embedding Predictive Architecture (JEPA). The architecture first pretrains a JEPA encoder on unlabeled GUI screenshots to map screen images into latent representations. A latent predictor conditioned on actions and regions then transforms the screen latent representation into a latent vector for each candidate action. Finally, a lightweight scoring head matches and scores the latent vectors of all candidate actions against the instruction in a single batched forward pass, without generating any text tokens throughout.

The direct benefit of this design is the elimination of decoding overhead. Traditional VLM agents, even when adopting candidate scoring strategies, still need to keep the large model in the inference loop; GUI-JEPA's scoring process, by contrast, involves only latent-space operations, drastically reducing parameter count.

Joint Embedding Predictive Architecture (JEPA) was proposed by Yann LeCun and others as a non-generative self-supervised learning framework. Unlike autoregressive models that reconstruct pixels or generate token by token, JEPA makes predictions in latent space: an encoder maps the input (e.g., an image) into a representation, and a predictor predicts the latent representation of the target part based on context. The training objective is to align the predicted representation with the target representation in embedding space, rather than reconstructing the original pixels. This design avoids modeling irrelevant details, yielding more compact and semantically richer representations, and requires no decoding at inference time. In the GUI setting, the JEPA encoder can therefore efficiently extract screen layout and widget information, providing a foundation for subsequent latent-space action scoring.

Candidate action scoring / reranking is a discriminative decision paradigm: the system pre-enumerates a finite set of candidate actions (e.g., gridded click coordinates or several swipe directions), and the model scores each candidate and selects the highest-scoring one. Compared with autoregressive generation, it transforms "open-ended sequence generation" into a "closed-set classification/ranking" problem, evaluating all candidates in parallel in a single forward pass and eliminating the serial overhead of token-by-token decoding. Its effectiveness depends heavily on the coverage of the candidate set: too coarse a set loses accuracy, while too fine a set increases computation. GUI-JEPA precisely migrates this paradigm from text scoring to latent-space scoring, thereby obtaining both the efficiency of candidate scoring and the compactness of latent representations.

Performance: The Speed-Accuracy Trade-off

On the AndroidControl benchmark, GUI-JEPA was compared with a Qwen2.5-VL-7B agent fine-tuned with the same inputs by the research team. Results show that on the same GPU, GUI-JEPA's inference speed is 2.8 to 4.0 times that of the latter, while using only 6% of the parameters to reach 86% to 92% of Qwen2.5-VL-7B's in-distribution accuracy.

This result means that for a large number of common GUI operation tasks, a lightweight latent-space policy can boost response speed several-fold and significantly reduce memory footprint with almost no sacrifice in usable accuracy. For scenarios requiring GUI agent deployment on-device or in resource-constrained environments, this trade-off is practically attractive.

AndroidControl is a public benchmark for Android GUI agents, containing a large number of real app screenshots and human operation trajectories. Tasks are given as natural language instructions and require the agent to output the next action (e.g., click, type, swipe). Its evaluation typically distinguishes "in-distribution" (task types and apps seen in the training distribution) from "out-of-distribution" settings, and examines both action type prediction and click location accuracy. Because its action space is mobile GUI and it provides instruction-screenshot-action triplets, it is often used to compare the step-by-step decision-making capabilities of different VLM agents. GUI-JEPA chose this benchmark to enable direct comparison of speed and accuracy under identical conditions with a 7B-level VLM baseline fine-tuned on the same inputs.

Key Finding: The Predictor Needs No Pretraining

A counterintuitive finding in the paper is worth noting: the latent predictor used for decision-making does not need to be pretrained on screen dynamics prediction tasks. Experiments show that a randomly initialized predictor performs comparably to one pretrained on screen dynamics. This indicates that the scoring head truly relies on the latent representation pathway provided by the JEPA encoder, rather than the dynamic knowledge learned by the predictor.

In other words, the value of JEPA pretraining lies mainly in the encoder's understanding of GUI screens, while the action-conditioned latent predictor is more like a lightweight adaptation layer learned on the spot for the decision task. This finding provides a basis for simplifying the training pipeline going forward—at least in this architecture, the predictor can skip the expensive pretraining stage.

The latent predictor and the scoring head serve different roles in this architecture. The latent predictor takes the screen latent representation along with candidate action and region conditions as input and outputs a conditioned latent vector for that candidate, effectively injecting "action intent" into the representation space; the scoring head is a lightweight matching module that computes similarity scores between these latent vectors and the instruction representation and ranks them. Both have far fewer parameters than a VLM and involve no text generation. Experiments show that the predictor works with random initialization, indicating that the information needed for decision-making mainly comes from the screen representation already learned by the JEPA encoder, and the predictor is more like a task-specific lightweight adaptation layer that can be quickly acquired during downstream training without expensive dynamic pretraining.

Limitations and Open Questions

GUI-JEPA has so far been validated only on AndroidControl, with a relatively limited action space and task types. The candidate action generation strategy, the adaptability of latent-space scoring to complex multi-step tasks, and generalization across platforms (e.g., desktop GUIs) are all questions that remain insufficiently answered. In addition, the 86% to 92% accuracy retention rate means that in accuracy-sensitive scenarios, large models still have their value.

Nevertheless, the direction of this work deserves attention: it demonstrates that GUI agents' action selection can break away from autoregressive decoding and be performed efficiently in latent space. The code is open-sourced on GitHub, providing a reproducible baseline for future research.

One-line Summary

GUI-JEPA replaces text decoding with JEPA latent-space candidate scoring, achieving several-fold speedup with far fewer parameters.

Paragraph Summary

GUI-JEPA proposes a mobile GUI agent policy that requires no text decoding: a JEPA encoder first maps screen screenshots into latent representations, then an action-conditioned latent predictor generates a latent vector for each candidate action, and finally a lightweight scoring head matches and scores them against the instruction in a single forward pass, without generating any tokens throughout. On AndroidControl, it is 2.8–4.0× faster than Qwen2.5-VL-7B fine-tuned on the same inputs, reaching 86%–92% of the latter's in-distribution accuracy with only 6% of the parameters. The paper also finds that the latent predictor needs no pretraining, with random initialization performing comparably to pretrained versions, indicating that decision-making relies mainly on the JEPA encoder's screen representation. The method has so far been validated on only a single benchmark, and candidate generation and cross-platform generalization remain to be explored.