GUI-JEPA
A mobile GUI agent policy based on the Joint Embedding Predictive Architecture (JEPA). It directly selects actions in latent space by scoring candidate actions without textual decoding, substantially improving inference speed while reducing parameter count.
Timeline (last 90 days)
GUI-JEPA proposes a mobile GUI agent policy that performs action selection directly in latent space without autoregressive text decoding.
The core idea of GUI-JEPA is to pre-enumerate discrete candidate actions, have the model score all candidates in a single batched forward pass and pick the highest-scoring one, generating no text tokens throughout.
The GUI-JEPA architecture comprises a JEPA encoder pretrained on unlabeled GUI screenshots, a latent predictor conditioned on actions and regions, and a lightweight scoring head.
On the AndroidControl benchmark, GUI-JEPA's inference speed is 2.8 to 4.0 times that of a Qwen2.5-VL-7B agent fine-tuned with the same input.
Using only about 6% of Qwen2.5-VL-7B's parameter count, GUI-JEPA reached 86% to 92% of the latter's in-distribution accuracy on AndroidControl.
Experiments show that a randomly initialized latent predictor and a latent predictor pretrained on screen dynamics perform comparably in GUI-JEPA, i.e., the latent predictor used for decision-making does not need pretraining.
GUI-JEPA has so far been validated only on AndroidControl, its action space and task types are relatively limited, and its cross-platform generalization ability has not yet been fully verified.
In accuracy-sensitive scenarios, 7B-scale large-model VLMs still have value relative to GUI-JEPA (an 86% to 92% accuracy retention rate implies an 8% to 14% accuracy loss).
All Facts (8)
GUI-JEPA proposes a mobile GUI agent policy that performs action selection directly in latent space without autoregressive text decoding.
50%UnverifiedThe core idea of GUI-JEPA is to pre-enumerate discrete candidate actions, have the model score all candidates in a single batched forward pass and pick the highest-scoring one, generating no text tokens throughout.
50%UnverifiedThe GUI-JEPA architecture comprises a JEPA encoder pretrained on unlabeled GUI screenshots, a latent predictor conditioned on actions and regions, and a lightweight scoring head.
50%UnverifiedOn the AndroidControl benchmark, GUI-JEPA's inference speed is 2.8 to 4.0 times that of a Qwen2.5-VL-7B agent fine-tuned with the same input.
50%UnverifiedUsing only about 6% of Qwen2.5-VL-7B's parameter count, GUI-JEPA reached 86% to 92% of the latter's in-distribution accuracy on AndroidControl.
50%UnverifiedExperiments show that a randomly initialized latent predictor and a latent predictor pretrained on screen dynamics perform comparably in GUI-JEPA, i.e., the latent predictor used for decision-making does not need pretraining.
50%UnverifiedGUI-JEPA has so far been validated only on AndroidControl, its action space and task types are relatively limited, and its cross-platform generalization ability has not yet been fully verified.
50%UnverifiedIn accuracy-sensitive scenarios, 7B-scale large-model VLMs still have value relative to GUI-JEPA (an 86% to 92% accuracy retention rate implies an 8% to 14% accuracy loss).
50%