LIBERO-PeRM: The First Personalized Robotic Manipulation Benchmark Released

For general-purpose robots to enter the home, being able to do the job is not enough—they need to learn to "do the job your way." For example, when pouring water, some people prefer the cup on the right hand side, while others prefer the left; when folding clothes, some want them sorted by color, others by material. This ability to satisfy personal preferences is becoming a key threshold for making robots practical.

The recently released LIBERO-PeRM benchmark is the first to separate "task completion" and "preference satisfaction" into two distinct evaluation dimensions. According to the project team, this is the first controlled benchmark for preference learning in embodied agents, aiming to establish quantifiable and reproducible evaluation standards for personalized robotic manipulation (PeRM).

Why Do We Need a Dedicated Personalization Benchmark?

Over the past few years, research in general-purpose robotic manipulation has advanced rapidly, with VLA (Vision-Language-Action) models continually setting new records in tasks like grasping, placing, and assembly. But these evaluations almost always ask only one question: was the task completed?

The problem is that task completion does not equal user satisfaction. A robot can put dishes into the dishwasher, but if you prefer them arranged in zones by size, it still hasn't done it "right." Existing benchmarks lack a mechanism to separate these two layers of capability, making it impossible to determine whether a model actually learned a preference or just happened to complete the task.

The core contribution of LIBERO-PeRM is a controlled formalized definition and taxonomy of preferences. It categorizes observable preferences in simulation into three types: motion preferences (e.g., which direction to approach an object from), skill preferences (e.g., grasping method), and user profile preferences (e.g., object placement position). This definition comes with several hard constraints: preferences must be expressible in language, must not alter the task goal, must be automatically evaluable, and must be reusable across tasks.

According to the project team, this ensures the benchmark tests "preference learning" itself, rather than padding the evaluation with new tasks.

VLA (Vision-Language-Action) models are a core paradigm in embodied intelligence in recent years. The basic idea is to unify visual information captured by cameras, natural language instructions, and robot action sequences into a single model, enabling it to "see the scene, understand the instruction, and produce actions." Representative examples include RT-2 and OpenVLA. Unlike traditional modular pipelines (perceive first, then plan, then control), VLA predicts actions end-to-end directly from raw input, giving it stronger generalization potential. But precisely because it is trained end-to-end, the boundary between task success and preference satisfaction becomes blurred—this is one reason LIBERO-PeRM needs a separately designed evaluation mechanism.

Supervised Fine-Tuning (SFT) is currently the most common method for adapting pre-trained models to downstream tasks: continuing to train model parameters on high-quality demonstration data for a specific task so that the output distribution shifts toward the target behavior. In preference learning scenarios, SFT's role is to let the model "see" preference patterns from demonstrations and imitate them. However, SFT has a structural contradiction: the deeper the model learns new data, the more it forgets existing capabilities—this is the so-called catastrophic forgetting. The paper's comparison between mixed training and sequential adaptation is essentially a trade-off between "learning new preferences" and "preserving old capabilities."

Five Questions, Five Task Suites

LIBERO-PeRM designs its evaluation system around five progressive questions:

• Under zero-shot conditions, can the policy follow preference instructions?
• From demonstrations, can the policy learn a single preference?
• As the number of demonstrations increases, how does preference acquisition efficiency change?
• Can learned preferences generalize to new layouts and new tasks?
• When multiple preferences conflict, can the policy compose them and handle them reasonably?

Building on the original LIBERO environments, the project team constructed a procedural data generation pipeline that automatically produces paired demonstrations "with preferences" and "without preferences." The final five task suites comprise 312 tasks, complete with high-quality demonstrations, rich metadata, and preference satisfaction metrics.

LIBERO itself is a widely used robotic manipulation benchmark platform, short for Lifelong Robot Learning Benchmark, providing multiple desktop manipulation environments, standardized task definitions, and demonstration data formats. Its features include high task diversity, stable simulation, and support for continual learning research. LIBERO-PeRM does not start from scratch but extends the original LIBERO environments with a preference dimension, thereby inheriting its task system and simulation infrastructure while adding preference definitions, a paired demonstration generation pipeline, and preference satisfaction metrics. This approach of "adding a layer on a mature platform" benefits result reproducibility and comparability with existing work.

Experimental Results: VLA's Personalization Capabilities Remain Weak

The paper's experimental conclusions are quite direct. Current mainstream VLA policies perform poorly in zero-shot personalization scenarios—they can complete tasks but struggle to follow preferences from instructions alone.

Supervised fine-tuning (SFT) makes preferences learnable, which is good news. But subsequent comparative experiments reveal more nuanced trade-offs:

Mixed training outperforms sequential adaptation in "no-preference retention" and also performs better than pure preference training when combined with in-context prompting, but it weakens motion preference learning. In other words, making a model both retain existing capabilities and learn new preferences currently has no free lunch.

As for preference generalization, decomposition, and conflict handling, the paper explicitly states these "remain difficult." This means preferences learned by models tend to be bound to specific scenarios, and easily fail when the layout changes or another preference is layered on top.

What This Means for the Industry

The value of this benchmark is not in achieving a new high score, but in transforming "personalization" from a vague product concept into a measurable research problem.

For teams building home robots, LIBERO-PeRM provides a ready-made evaluation toolkit to test their models' true level of preference learning, rather than just looking at task success rates. For researchers, the five questions constitute a clear roadmap: from zero-shot following to multi-preference composition, each step has well-defined failure modes and challenges to overcome.

The code, dataset, and demonstrations are publicly available on the project website. The field of personalized robotic manipulation finally has its first proper measuring stick.

One-line Summary

LIBERO-PeRM is the first to separate task completion from preference satisfaction in evaluation, revealing that current VLA models have weak personalization capabilities.

Paragraph Summary

LIBERO-PeRM is the first controlled benchmark for preference learning in embodied agents, separating "task completion" from "preference satisfaction" into distinct evaluations. It proposes three formalized preference categories—motion, skill, and user profile preferences—and builds five task suites with 312 tasks around five progressive questions: zero-shot following, single-preference learning, acquisition efficiency, generalization, and conflict handling. Experiments show that current mainstream VLA policies perform poorly in zero-shot personalization; SFT can make preferences learnable, but mixed training weakens motion preference learning, and preference generalization, decomposition, and conflict handling remain difficult. The benchmark's value lies in turning personalization from a vague concept into a measurable research problem and providing a ready-made evaluation toolkit.