Fatal Flaw in JEPA World Models: Action Ranking Failures Masked by Replanning

The Core Assumption of Reward-Free World Models Under Challenge

Reward-free latent world models evaluate candidate actions by computing distances in a frozen latent space: if a particular action's predicted future embedding is closer to the goal embedding, that action is prioritized. This planning strategy relies on a key implicit assumption — proximity in latent space can be used for action ranking.

However, new research challenges this assumption. The research team developed ARC-Bench, a benchmark specifically designed to audit whether JEPA (Joint-Embedding Predictive Architecture)-style world models can correctly rank candidate actions. The results show that this foundational assumption systematically fails across multiple task scenarios.

Systematic Action Ranking Failures

The research team conducted a comprehensive audit of officially released JEPA-WM checkpoints, covering both navigation and manipulation control tasks. The experimental results revealed a serious problem:

In the official manipulation task audit, the highest-scoring candidate action was almost always suboptimal. The same inversion phenomenon also appeared in maze navigation tasks. This is not an isolated case — it is a systematic flaw.

To rule out the influence of the visual encoder, the team replaced DINOv2 with video-pretrained V-JEPA 1 and V-JEPA 2 encoders (ViT-L/ViT-G scale), and found that the flaw persisted. This indicates that the root cause lies in the representational capacity of the JEPA architecture itself, not in the specific choice of visual encoder.

The study also ruled out simpler explanations such as model provenance, insufficient training, backbone controls with matched budgets, and metric cyclicity controls — suggesting the problem is more fundamental and deep-rooted.

How Replanning Masks the Core Flaw

An even more critical finding is that closed-loop replanning masks this flaw. This explains why such a serious issue has gone undetected.

When the research team reduced the planner's replanning frequency, success rates in both navigation and manipulation tasks collapsed dramatically. Further analysis showed that episodes relying on frequent replanning to succeed were precisely the ones enriched with severe first-plan ranking failures in the PointMaze first-plan diagnostics.

This reveals a key mechanism: the system compensates for inadequate ranking ability in latent representations through continuous error correction. Each replanning step serves as a correction opportunity, allowing the model to gradually approach the right direction despite an incorrect initial ranking. As a result, closed-loop success rates systematically overestimate the rankability of frozen latent representations.

ARC-Bench: Directly Auditing Action Ranking Capability

ARC-Bench provides a leak-free, fixed-candidate testing protocol specifically designed to measure whether frozen JEPA-style objective functions can correctly rank candidate actions. Key features of this benchmark include:

1. Fixed candidate sets: Eliminates evaluation uncertainty introduced by dynamically generated candidate actions
2. No information leakage: Ensures fairness and reliability of the tests
3. Direct auditing: Measures action ranking capability directly, rather than relying on indirect metrics like closed-loop success rates

Traditional evaluations tend to focus only on final task success rates while overlooking the quality of intermediate planning steps. ARC-Bench fills this gap and offers a new perspective for evaluating world models.

Implications for Reinforcement Learning Planning Systems

This research has critical implications for building reliable world models and planning systems:

Evaluation methodology needs upgrading: Closed-loop success rate has serious limitations as a sole evaluation metric. Researchers need to develop more fine-grained diagnostic tools that directly test whether a model's core capability assumptions actually hold.

The JEPA architecture needs re-examination: Relying solely on latent distances to guide action selection is insufficient. New representation learning objectives or planning algorithms need to be explored.

The hidden cost of replanning strategies: While frequent replanning can improve task success rates, it introduces computational overhead and masks underlying representational issues. In practice, over-reliance on replanning may cause systems to fail in scenarios where decisions cannot be frequently updated.

Potential Directions for Improving World Models

In response to the discovered issues, the research community can explore the following directions:

• Improved representation learning: Design new self-supervised learning objectives that endow latent spaces with inherent action ranking capabilities
• Hybrid planning architectures: Combine distance-based planning with value-based planning, leveraging the strengths of each
• Enhanced interpretability: Develop tools to visualize and understand the structure of latent spaces and identify patterns of ranking failure

This research reminds us that even seemingly successful AI systems may harbor fundamental flaws in their internal mechanisms. Only through rigorous auditing and diagnostics can we truly advance the technology.