CALVIN
A benchmark for evaluating long-horizon robotic manipulation tasks, requiring a policy to complete multiple subtasks under a continuous instruction sequence and recording the average completed chain length; widely used in goal-conditioned manipulation research.
Timeline (last 90 days)
Tests on the CALVIN benchmark show that when leading image editors on image-editing quality leaderboards (such as Gemini 3 Pro Image and GPT Image 1.5) are used as control targets, their success rates are lower than those of fine-tuned open-source models.
The above phenomenon of editing quality being decoupled from success rate is verified on both the CALVIN and SIMPLER-Bridge benchmarks.
SGE-Goal, fine-tuned on FLUX.1 Kontext and FLUX.2 klein, ranks first on CALVIN.
Meta-World provides 50 tabletop manipulation tasks of varying difficulty, RLBench includes 100 tasks and supports multiple observation modes, and CALVIN focuses on language-conditioned long-horizon manipulation task chains — these are the main simulation evaluation benchmarks in the field of robotic manipulation.
Benchmark platforms such as Meta-World, RLBench, and CALVIN provide standardized multi-task evaluation environments.
All Facts (5)
Tests on the CALVIN benchmark show that when leading image editors on image-editing quality leaderboards (such as Gemini 3 Pro Image and GPT Image 1.5) are used as control targets, their success rates are lower than those of fine-tuned open-source models.
50%UnverifiedThe above phenomenon of editing quality being decoupled from success rate is verified on both the CALVIN and SIMPLER-Bridge benchmarks.
50%UnverifiedSGE-Goal, fine-tuned on FLUX.1 Kontext and FLUX.2 klein, ranks first on CALVIN.
50%UnverifiedMeta-World provides 50 tabletop manipulation tasks of varying difficulty, RLBench includes 100 tasks and supports multiple observation modes, and CALVIN focuses on language-conditioned long-horizon manipulation task chains — these are the main simulation evaluation benchmarks in the field of robotic manipulation.
50%UnverifiedBenchmark platforms such as Meta-World, RLBench, and CALVIN provide standardized multi-task evaluation environments.
50%