Goal Images Are Control Commands: How Spatially Grounded Goal Image Synthesis Reshapes Robotic Manipulation

A Widely Accepted Assumption Is Slowing Down Robot Manipulation Research

The core input of a goal-conditioned manipulation policy is a goal image — it depicts the desired scene after the manipulation is complete, synthesized from the current observation and a language instruction. The research community has long operated on a default logic: the better the image editing quality, the more effective the goal image. As a result, goal generators have mostly been designed and compared according to general-purpose image editing metrics.

This paper directly challenges that assumption. After testing two independent goal-conditioned policies on the CALVIN benchmark, the authors found that editors topping the image editing quality leaderboards (such as Gemini 3 Pro Image and GPT Image 1.5) actually underperform much smaller fine-tuned open-source models as control targets. More critically, this gap is dramatically amplified on multi-step long-horizon tasks — precisely the scenarios real robots face.

Why Does Editing Quality Decouple from Policy Success Rate?

Dimension-by-Dimension Breakdown: Manipulation Adherence Is What Matters

The paper conducts a dimension-by-dimension analysis, and the conclusion is quite clear: downstream policy success is almost entirely determined by manipulation adherence — that is, whether the right object is selected and whether the correct final state is generated. The Spearman correlation coefficient between this dimension and success rate is approximately 0.9.

By contrast, photorealism and aesthetic fidelity have correlations with success rate close to zero. And these two are exactly the strengths of large-scale closed-source editors. The paper validates this dissociation phenomenon on both CALVIN and the real-world-aligned SIMPLER-Bridge benchmark.

The implication for the research community is direct: using editing quality metrics like FID and CLIP Score to select goal generators may well select models that are precisely unsuitable as control targets.

The Spearman correlation coefficient is a non-parametric statistic used to measure the monotonic correlation between two variables, with values ranging from -1 to 1. Unlike the Pearson correlation coefficient, it does not require variables to follow a normal distribution or exhibit a linear relationship; instead, it is computed based on the ranks of the variables. In robot manipulation research, the Spearman correlation coefficient is often used as an analytical tool when assessing the strength of association between an intermediate metric (such as image editing quality or manipulation adherence) and the final policy success rate. A coefficient close to 1 indicates that the two rankings change in highly consistent directions, while a value close to 0 indicates that they are essentially independent. The paper reports a Spearman correlation coefficient of approximately 0.9 between manipulation adherence and success rate, meaning that ranking by this dimension can almost accurately predict the ranking of policy success rates; meanwhile, the correlation coefficients for photorealism and aesthetic fidelity are close to zero, indicating that they are statistically unrelated to success rate.

The Penalty on Long-Horizon Tasks Is Even Heavier

On single-step tasks, the misleading nature of editing quality is relatively mild; once multi-step chained manipulation is involved, errors accumulate. The paper points out that the success rate of the best closed-source editor on the longest five-step chains is roughly seven times lower than that of its SGE-Goal fine-tuned model. This shows that a goal image is not just an image that "looks right" — it is a control signal that requires precise spatial grounding.

Chained tasks (long-horizon chained tasks) refer to manipulation settings in which a robot must sequentially complete a series of subgoals to achieve the final task, such as a process composed of multiple steps like "open the drawer → take out the object → place it on the table → close the drawer." Unlike single-step tasks, the input observation at each step in a chained task depends on the execution result of the previous step, so errors in early steps propagate backward and accumulate, forming compound errors. The CALVIN benchmark is designed precisely to evaluate this kind of long-horizon manipulation: it requires the policy to complete multiple subtasks under a continuous instruction sequence and records the average completed chain length. The paper emphasizes that if a goal image only "looks right" but is not spatially precise enough, it may cause only a single failure in a single-step task, but in a five-step chain, a deviation in one subgoal may cause all subsequent steps to drift — which is exactly why closed-source editors see success rates plummet on long-chain tasks while the SGE-Goal fine-tuned model remains stable.

SGE-Goal: Localize First, Then Generate

The paper proposes SGE-Goal, a spatially grounded decomposed editing framework. Its core idea is to split goal image synthesis into two steps: first predict the initial and final 2D edit region masks, then generate the goal image based on the masks.

This decomposition turns open-source base models (FLUX.1 Kontext, FLUX.2 klein) into the strongest goal generators in the experiments: ranking first on CALVIN and second only to Gemini 3 Pro Image on SIMPLER-Bridge. And the base models fine-tuned with SGE-Goal far exceed the best closed-source editors in stability on chained tasks.

The paper also releases code, datasets, checkpoints, and a human-aligned diagnostic evaluator, which is a substantive contribution to community reproduction and follow-up research.

An "edit region mask" in image editing is a binary or soft-weight map of the same size as the input image, used to indicate where and to what extent an editing operation should occur. In decomposed goal image synthesis, the model first predicts "the regions in the initial state that need to be changed" and "the regions corresponding to the final goal state," and then uses these masks as spatial constraints to guide the generative model to synthesize content only within these regions. The advantage of this approach is that it decouples "where to change" from "what to change it to": mask prediction is responsible for spatial grounding, while the generative model is responsible for appearance synthesis. Compared with end-to-end direct goal image generation, masks provide an explicit spatial prior, which can significantly reduce the risk of the generative model hallucinating or altering irrelevant regions, and also makes manipulation adherence — that is, whether the right object is selected and whether the correct final state is reached — easier to guarantee.

Implications for Goal Generator Selection

The core argument of this work can be summarized as follows: goal generators must be selected according to downstream utility, not according to editing quality leaderboards. For researchers and engineers working on robot policy learning, this means the evaluation pipeline needs adjustment — use policy success rate as the first criterion for goal generator selection, rather than first looking at image editing metrics and assuming they are effective.

Spatially grounded decomposed editing provides a viable path: rather than pursuing more photorealistic images, let the model learn "where to change." This direction is consistent with the current emphasis on spatial reasoning capabilities in embodied intelligence, and also suggests that the evaluation systems of goal image synthesis and visual editing need to diverge.

The limitation of the paper is that SGE-Goal depends on the editing capability of the base model, and the decomposed framework introduces an additional source of mask prediction error. However, the results show that this error is far smaller than the loss of control signal caused by pursuing photorealism.

One-line Summary

Goal image editing quality is decoupled from policy success rate; manipulation adherence is what matters, and SGE-Goal's decomposed editing significantly improves long-horizon manipulation.

Paragraph Summary

The paper challenges the default assumption in robot manipulation research that "the better the image editing quality, the more effective the goal image." On the CALVIN benchmark, the authors find that closed-source editors ranking high on editing quality leaderboards (such as Gemini 3 Pro Image and GPT Image 1.5) actually underperform fine-tuned small open-source models as control targets, and the gap is dramatically amplified on multi-step long-horizon tasks. A dimension-by-dimension analysis shows that downstream success is almost entirely determined by manipulation adherence (Spearman correlation coefficient approximately 0.9), while photorealism and aesthetic fidelity have correlations with success rate close to zero. To address this, the paper proposes SGE-Goal — a decomposed framework that first predicts 2D edit region masks and then generates the goal image based on the masks — turning open-source base models into the strongest goal generators on CALVIN and achieving roughly seven times the success rate of the best closed-source editor on five-step chain tasks.