CoDA: Co-Evolving Data Engine and Action Policy — A New Paradigm for Robot Training

In robot policy training, generative simulation can automatically construct environments, but a physically valid environment is not necessarily useful for training. What is truly scarce is environments that precisely expose a policy's failure modes, along with effective supervision that helps the policy overcome those failures. To address this core tension, researchers propose the CoDA framework, which enables a simulated data engine and an action policy to co-evolve through a shared heterogeneous memory.

Core Mechanism: Bidirectional Evolution Driven by Heterogeneous Memory

The key design of CoDA lies in a shared heterogeneous memory that simultaneously stores three types of information: environment construction records, policy failure cases, and learning progress metrics. A vision-language critic diagnoses policy failures, which after verification are compiled into two types of programs — generation programs used to reproduce difficult scenarios, and training programs used to help the policy overcome those difficulties.

This design makes environment generation no longer random sampling, but purposefully targeted at the policy's current weaknesses. Meanwhile, the measurement of learning progress in turn guides subsequent generation and program optimization, forming a closed loop.

Action-chunk value learning is a class of methods in recent robot policy learning. Unlike outputting a single-step action per frame, it lets the policy predict a sequence of actions within a time window (i.e., an action chunk) at once, and estimates value for the chunk as a whole. This has two benefits: first, it reduces decision frequency, alleviating the gradual accumulation of errors in long-horizon tasks; second, it allows the supervision signal to act on more semantically meaningful "action units." CoDA's Text2potential applies potential-based shaping precisely to this chunk-level value, so that the staged supervision signals derived from diagnosis can be aligned to specific action chunks rather than just single-step actions, thereby more efficiently correcting the policy's entire behavior in difficult scenarios.

Vision-language critic refers to a diagnostic module based on a vision-language model (VLM). It simultaneously receives visual observations and task description text, performing semantic-level evaluation of the policy's execution process — for example, judging failure modes such as "the object slips during grasping" or "path planning takes a detour," rather than merely outputting a scalar reward. Compared with traditional reward models that only provide a numerical value, a VLM critic can explain the reasons for failure in natural language, which provides a parsable intermediate representation for subsequently compiling diagnoses into "generation programs" (reproducing failure scenarios) and "training programs" (targeted reinforcement). CoDA adds a verification step on top of this to prevent misjudgments by the critic from directly contaminating the training process.

Text2potential: Converting Diagnosis into Dense Supervision

A difficult environment alone is not enough; the policy needs sufficiently dense supervision signals to learn effectively. CoDA proposes the Text2potential method, which converts the critic's diagnostic text into staged potentials (staged potentials), used for action-chunk value learning.

Under the standard potential-based shaping assumption, this method provides dense supervision while preserving the task's optimal policy — that is, it does not distort the original optimal solution by introducing shaping. This point is crucial for the stability of policy learning.

Potential-based reward shaping is a framework proposed in reinforcement learning to alleviate the problem of sparse rewards. Its core theorem states that if the additional reward function F can be written as F(s, a, s') = γΦ(s') − Φ(s) (where Φ is a state potential function and γ is the discount factor), then introducing this shaping will not change the set of optimal policies of the original MDP. CoDA compiles the critic's diagnostic text into staged potentials, essentially leveraging this theorem — encoding the semantic diagnosis of "how far the current state is from the goal" as a numerical potential difference, thereby providing a gradient signal to each action step without distorting the task's optimal solution. Without this constraint, artificially set shaping rewards can easily induce reward hacking, where the policy pursues a shaping bypass rather than truly completing the task.

Experimental Results: Significant Improvement in OOD Success Rate

In tests on four manipulation task families, CoDA raised the out-of-distribution (OOD) success rate from 5.3% after supervised fine-tuning to 27.6%, and under the same generation budget, outperformed the strongest closed-loop baseline by 9.5 percentage points.

On the LIBERO-Pro benchmark, after 15 rounds of generation and training, the success rate under positional perturbations increased from 25% to 84%, and under instruction perturbations it jumped from 1% to 76%. This result shows that targeted environment generation and diagnosis-driven supervision yield significant improvements in policy robustness.

Analysis: Why Co-Evolution Is More Effective

The traditional approach separates environment generation from policy training, resulting in generated environments that, although physically valid, are disconnected from the policy's current needs. CoDA's value lies in directly linking "where the policy fails" with "what environment should be generated," and through Text2potential, converting failure diagnoses into usable learning signals.

This co-evolution approach essentially transforms the data engine from a role of "passively providing data" to one of "actively diagnosing and reinforcing the policy's weaknesses." For fields like robot learning where data is scarce and trial-and-error costs are high, this paradigm is expected to reduce ineffective environment construction and improve training efficiency.

One-line Summary

CoDA uses a shared heterogeneous memory to let a simulated data engine and robot policy co-evolve, raising OOD success rate from 5.3% to 27.6%.

Paragraph Summary

This article introduces the CoDA framework, addressing the waste of environments that are "physically valid but useless for training" in generative simulation by enabling a data engine and action policy to co-evolve through a shared heterogeneous memory. The memory simultaneously stores environment construction records, policy failure cases, and learning progress metrics; a vision-language critic diagnoses failures and compiles them into generation programs and training programs, used respectively to reproduce difficult scenarios and reinforce the policy's weaknesses. The accompanying Text2potential method converts diagnostic text into staged potentials for action-chunk value learning, providing dense supervision while preserving the optimal policy. Experiments show that the OOD success rate increased from 5.3% to 27.6%, surpassing the strongest closed-loop baseline by 9.5 percentage points; on LIBERO-Pro, after 15 iterations, the success rate under positional perturbations rose from 25% to 84%, and under instruction perturbations from 1% to 76%.