MC-EvoBench: Scrutinizing Experience-Driven Self-Evolution in Minecraft Agents

Can embodied agents genuinely convert the experiences accumulated through repeated interactions into reusable capabilities that transfer across tasks and environments? This has long been an unresolved question in recursive self-improvement research. A saved memory or a successful final action is not sufficient evidence that an agent has truly acquired a transferable skill. MC-EvoBench addresses exactly this core question by constructing a controllable, reproducible evaluation benchmark within the Minecraft environment.

The Core Problem: Genuine Capability Gains vs. Resource Transfers

The biggest trap in evaluating an embodied agent's self-improvement is conflating several fundamentally different phenomena. When an agent performs better on a target task, it may not be because it learned new capabilities — it could be carrying over resources accumulated in the source task, benefiting from unequal training exposure, or simply having early execution failures mask later performance.

In other words, without careful controls, it is easy to mistake "bringing the tools along" for "learning how to make the tools." The research team emphasizes that rigorous evaluation must strictly disentangle genuine capability change from cross-task resource transfer, imbalanced source exposure, and early execution bottlenecks — otherwise, so-called "self-evolution" is nothing more than a statistical illusion.

"Recursive Self-Improvement" refers to the process by which an agent automatically optimizes its own strategies or knowledge representations through sustained interaction with the environment, leading to better performance on subsequent tasks. In embodied intelligence, this typically involves the agent encoding past experiences into some reusable form — such as code snippets, skill functions, or memory summaries — and retrieving them when facing new tasks. LLM-based Minecraft agents like Voyager and DEPS are representative works in this direction; they abstract successfully executed behaviors into a "skill library" that continuously expands. However, such work generally lacks rigorous controlled experiments, making it difficult to distinguish "the agent genuinely acquired transferable skills" from "the agent simply reused old resources or code in a new context." MC-EvoBench was designed specifically to fill this evaluation gap.

How MC-EvoBench Designs the Controlled Experiments

MC-EvoBench uses a fixed task catalog containing 19 registered evolution chains — 12 primary evaluation chains and 7 for calibration and probing. It also includes three graph-aligned paired probe families specifically designed to examine capability transfer.

Several key design principles are worth noting:

• Task state graph representation: Each task is modeled as a task state graph with phase-level environment validators to confirm whether the agent has truly reached a given intermediate state.
• Source–target role matching: Tasks are assigned matching source and target roles, with the target world reset before the agent enters the target task.
• Strict boundary control: Only one explicitly declared agent artifact is permitted to cross the source–target boundary, eliminating interference from covert resource transfers.

For each probe, the study establishes four source conditions for comparison — no-prior, related, unrelated, and shuffled — running five random seeds under a matched environment step budget and reporting three metrics: TSR, WGP, and VMP, along with complete provenance records. This design turns "does relevant experience actually help?" into a quantifiably testable proposition.

TSR, WGP, and VMP are the three core evaluation metrics defined by MC-EvoBench, each corresponding to a different dimension of capability measurement. TSR (Task Success Rate) measures the proportion of target tasks the agent ultimately completes — the most direct outcome metric. WGP (Weighted Goal Progress) performs a weighted calculation based on completion across phases in the task state graph, capturing how far the agent progressed even if it did not finish the entire task. VMP (Verified Milestone Progress) further relies on phase-level environment validators, counting only intermediate states confirmed by external environment verification rather than the agent's self-report, preventing agents from "claiming completion without actually completing" from inflating scores. Used together, the three metrics provide a three-dimensional portrait of agent performance across outcome, process, and credibility.

Key Finding: Relevant Experience Does Not Produce Stable Transfer

The experimental results deliver a sobering conclusion. The benchmark reveals a consistent direct capability gap between controller variants and planner variants, confirming that planning ability itself does influence performance.

But the more striking finding concerns transfer: relevant experience did not produce stable capability transfer. Specifically — it did not improve performance on any tool probe; it yielded only marginal gains on wood-type tasks, with results highly dependent on individual seed outcomes; and all confidence intervals include zero, meaning the hypothesis of "no effect" cannot be rejected on statistical grounds.

This finding serves as a wake-up call for much current research claiming "experience-driven self-improvement": seemingly relevant historical experience does not necessarily translate into generalizable capability.

Hidden Bottlenecks Revealed by Phase Auditing

Beyond the transfer findings, phase-level auditing exposes another easily overlooked problem. Early wood gathering and crafting steps frequently become bottlenecks — when agents get stuck on these prerequisite steps, there is simply no opportunity to observe more advanced downstream capabilities.

This means that in many evaluations, the conclusion that "the agent didn't learn a certain capability" may actually be an artifact of "the agent never reached that stage." MC-EvoBench uses phase validators to separately identify these execution failures, preventing them from contaminating assessments of capability acquisition. This fine-grained attribution is precisely what coarser evaluation approaches have been missing.

The reason wood gathering and crafting become systematic bottlenecks in Minecraft is closely tied to the game's crafting tree structure. The vast majority of advanced items — stone pickaxes, shovels, iron ingots — depend on wooden tools or a crafting table as prerequisites, forming a synthesis chain that must be unlocked in sequence. This means that if an agent fails at the earliest steps of "chop tree → craft planks → build crafting table → craft wooden pickaxe," all subsequent higher-level skills can never be triggered. In coarse end-to-end evaluations, these prerequisite failures are uniformly recorded as "task failure" and cannot be distinguished from "the agent reached the relevant stage but the skill itself was insufficient." Phase validators address this by placing independent environment snapshot checkpoints at each intermediate state, precisely separating "never reached that stage" from "reached it but failed," enabling researchers to more accurately locate where capability bottlenecks actually lie.

Value and Significance

MC-EvoBench's contribution is not a stronger agent, but rather a controllable, reproducible, and provenance-tracked testing platform for rigorously examining the durability and transferability of embodied capabilities. It pulls "self-evolution" back from a vague narrative into a falsifiable scientific proposition.

For researchers, the significance of this kind of benchmark lies in raising the evidentiary bar: the next time a system claims to have learned reusable capabilities from experience, we now have a measuring stick to determine whether it has truly evolved — or simply redeployed old resources in a new setting. The relevant code and data have been open-sourced on GitHub (github.com/xzw-ustc/mc-bench-open-source), providing a directly reusable experimental framework for future work.