MC-EvoBench
A benchmark for evaluating the experience-driven self-evolution capabilities of embodied agents in the Minecraft environment. It uses controlled experiments to distinguish genuine capability improvement from cross-task resource transfer, employing three metrics: TSR, WGP, and VMP.
Timeline (last 90 days)
MC-EvoBench is a controllable and reproducible evaluation benchmark built in the Minecraft environment for assessing agents' experience-driven self-evolution
MC-EvoBench uses a fixed task catalog containing 19 registered evolution chains, of which 12 are primary evaluation chains and 7 are used for calibration and probing
MC-EvoBench designed three groups of figure-aligned paired probe families to examine capability transfer
MC-EvoBench models each task as a task state graph and defines stage-level environment validators to confirm whether an agent has reached a certain intermediate state
MC-EvoBench only permits one explicitly declared agent artifact to cross the source–target boundary, in order to prevent resource transportation from interfering with evaluation
MC-EvoBench sets four source condition comparisons for each probe—no prior, relevant, irrelevant, and shuffled—and runs with five random seeds under a matched environment step budget
MC-EvoBench reports three core evaluation metrics: TSR (Task Success Rate), WGP (Weighted Goal Progress), and VMP (Verified Milestone Progress)
Experiments revealed a consistent direct capability gap between the controller variant and the variant with a planner, indicating that planning capability affects performance
Experiments found that relevant experience did not produce stable capability transfer, did not improve performance on any of the tool probes, and all confidence intervals contained zero
Marginal gains were observed on the wood type task, but these were highly dependent on the chance results of a single seed
2 more timeline events
All Facts (12)
MC-EvoBench is a controllable and reproducible evaluation benchmark built in the Minecraft environment for assessing agents' experience-driven self-evolution
50%UnverifiedMC-EvoBench uses a fixed task catalog containing 19 registered evolution chains, of which 12 are primary evaluation chains and 7 are used for calibration and probing
50%UnverifiedMC-EvoBench designed three groups of figure-aligned paired probe families to examine capability transfer
50%UnverifiedMC-EvoBench models each task as a task state graph and defines stage-level environment validators to confirm whether an agent has reached a certain intermediate state
50%UnverifiedMC-EvoBench only permits one explicitly declared agent artifact to cross the source–target boundary, in order to prevent resource transportation from interfering with evaluation
50%UnverifiedMC-EvoBench sets four source condition comparisons for each probe—no prior, relevant, irrelevant, and shuffled—and runs with five random seeds under a matched environment step budget
50%UnverifiedMC-EvoBench reports three core evaluation metrics: TSR (Task Success Rate), WGP (Weighted Goal Progress), and VMP (Verified Milestone Progress)
50%UnverifiedExperiments revealed a consistent direct capability gap between the controller variant and the variant with a planner, indicating that planning capability affects performance
50%UnverifiedExperiments found that relevant experience did not produce stable capability transfer, did not improve performance on any of the tool probes, and all confidence intervals contained zero
50%UnverifiedMarginal gains were observed on the wood type task, but these were highly dependent on the chance results of a single seed
50%UnverifiedStage-level audits found that early wood harvesting and crafting steps frequently became bottlenecks, as agents getting stuck in prerequisite steps prevented higher-level capabilities from being observed
50%UnverifiedThe relevant code and data for MC-EvoBench have been open-sourced on GitHub at github.com/xzw-ustc/mc-bench-open-source
50%