AnyGoal Benchmark: World Model Generalization Faces Compositional Challenges

AnyGoal Benchmark: World Model Generalization Faces Compositional Challenges

The core goal of world models is to capture environment dynamics, enabling agents to plan actions through simulation. However, for such planning to be reliable, the learned dynamics must generalize to state-action combinations beyond expert trajectories. Although the relationship between data coverage, dynamic generalization, and downstream planning performance is crucial, it has lacked systematic study. The recently proposed AnyGoal benchmark delves into this issue from the perspective of compositional generalization: can world models recombine familiar state and action factors into effective dynamics?

Compositional Generalization: A Key Test for World Model Generalization

The core idea of AnyGoal is to construct "goal-shifted" task variants—tasks built on widely used world model environments and accompanied by an open-source library. Each task is designed to ensure that if the world model truly learns the underlying dynamics, it should be able to complete it. Specifically, individual state factors and actions remain within the training support, but their combinations rarely appear in expert trajectories. This is akin to asking a model that has only seen "red squares" and "blue circles" to understand "red circles" or "blue squares," testing the model's ability to generalize over factor combinations rather than simple memorization.

This design avoids extreme cases requiring entirely new states or actions and precisely targets the weak spot of compositional generalization. Researchers note that current models "consistently fail" under such recombination, revealing a fundamental limitation of world models when going beyond expert demonstrations.

World models are internal representations of environment dynamics, typically trained on large amounts of trajectory data, capable of predicting the next state given the current state and action. In model-based planning, agents use world models to simulate the consequences of different action sequences and select the optimal action. The quality of the world model directly determines planning reliability, but its generalization ability—especially performance on state-action combinations not covered by training data—remains a core challenge.

Compositional generalization refers to a model's ability to recombine independent factors seen during training (such as color, shape, action type) into new combinations. In visual reasoning and language understanding, compositional generalization has been widely studied, but in the realm of world models, it requires the model to understand the interaction dynamics between factors rather than simply memorizing the joint distribution. For example, if training data only contains "red squares" and "blue circles," the model needs to infer the physical behavior of "red circles," testing its capture of underlying causal structure.

Widespread Failure Across Multiple Environments and Model Families

According to the paper, the study tested multiple environments and various world model families, consistently finding that current models perform poorly on compositional generalization tasks. This failure is not a defect of individual models but a widespread phenomenon across model families. Researchers further analyzed dynamic failures and representation failures, involving different interaction mechanisms and evaluation metrics, attempting to pinpoint where models can generalize and where they still face challenges.

This finding has important implications: even if a world model performs well within the training distribution, its reliability drops significantly once it needs to recombine known elements to plan for new tasks. For agents relying on world models for decision-making, this means they may face unforeseen risks when deployed in the real world.

Goal-shifted tasks are an experimental design for evaluating compositional generalization: by changing task goals (e.g., recombining factors in the goal state), they generate planning tasks that are out-of-distribution but theoretically solvable. Such tasks ensure that individual state factors and actions remain within the support, but combinations rarely appear in expert trajectories. Goal-shifted tasks precisely test whether the model has learned the interaction rules between factors, rather than merely fitting the trajectory distribution of expert demonstrations.

Benchmark and Open-Source Library: Advancing Research on Generalizable World Models

AnyGoal is not only a diagnostic tool but also a public benchmark and open-source library aimed at providing a foundation for evaluating progress in generalizing world models to new planning tasks. Through a standardized task suite and evaluation protocol, researchers can more systematically compare the generalization capabilities of different methods, thereby advancing the field toward truly generalizable world models.

The introduction of this benchmark fills the research gap regarding the relationship between data coverage and planning performance. In the future, using AnyGoal for model development and evaluation is expected to help researchers design architectures and training strategies that better capture compositional dynamics, ultimately achieving world models that can reliably plan under unfamiliar but reasonable state-action combinations.

For more technical details and open-source resources, please refer to the original paper and project page.

One-line Summary

AnyGoal benchmark reveals that world models generally fail on compositional generalization tasks.

Paragraph Summary

The AnyGoal benchmark systematically tests the generalization ability of world models over state-action factor combinations through goal-shifted task variants. Experiments show that current world models consistently fail across multiple environments and model families, unable to reliably recombine familiar factors to plan for new tasks. This research fills the gap in understanding the relationship between data coverage, dynamic generalization, and downstream planning performance, and provides an open-source benchmark and standardized evaluation tools to advance research on generalizable world models.