WorldOracle
A benchmark for evaluating action-conditioned world models. It compares model-predicted rollouts frame by frame against recordings of a real robot executing the same action sequence, measuring physical fidelity of predictions along four dimensions: physical outcomes, spatial alignment, visual fidelity and cross-view action consistency.
Timeline (last 90 days)
WorldOracle is a benchmark that compares, frame by frame, the predicted rollouts of an action-conditioned world model against footage of a real robot executing the same action sequence.
WorldOracle's comparison baseline is the execution results in the real physical world, not another generated video.
WorldOracle's evaluation dimensions fall into four categories: physical outcomes, spatial alignment during interaction, visual fidelity of the interaction region, and action consistency across viewpoints.
In WorldOracle, if an action causes a real grasp to fail, the model is judged as correctly predicting only if it reproduces that failure; if the model generates a pick that looks successful, it is judged wrong on the outcome dimension.
At the end of a Reddit post, the WorldOracle project team publicly solicited feedback from researchers in robot learning and learned dynamics, asking what the evaluation covers and what they would additionally like to see tested.
WorldOracle is positioned as a controlled benchmark for action-conditioned world models rather than a general video generation benchmark, and it cares about the credibility of models as internal simulators for robots.
All Facts (6)
WorldOracle is a benchmark that compares, frame by frame, the predicted rollouts of an action-conditioned world model against footage of a real robot executing the same action sequence.
50%UnverifiedWorldOracle's comparison baseline is the execution results in the real physical world, not another generated video.
50%UnverifiedWorldOracle's evaluation dimensions fall into four categories: physical outcomes, spatial alignment during interaction, visual fidelity of the interaction region, and action consistency across viewpoints.
50%UnverifiedIn WorldOracle, if an action causes a real grasp to fail, the model is judged as correctly predicting only if it reproduces that failure; if the model generates a pick that looks successful, it is judged wrong on the outcome dimension.
50%UnverifiedWorldOracle is positioned as a controlled benchmark for action-conditioned world models rather than a general video generation benchmark, and it cares about the credibility of models as internal simulators for robots.
50%UnverifiedAt the end of a Reddit post, the WorldOracle project team publicly solicited feedback from researchers in robot learning and learned dynamics, asking what the evaluation covers and what they would additionally like to see tested.
50%