LIBERO-RIG
A benchmark for VLA models focused on robustness to perturbations in robot hardware and setup. It keeps LIBERO tasks unchanged and covers 11 perturbation dimensions, including camera mounting and optical parameters, calibration offsets, actuation gains, control frequency, proprioceptive noise, base pose, table height, and gripper pad state. Each dimension has five severity levels, combining into 210 test conditions across four LIBERO suites, with 500 episodes per condition.
Timeline (last 90 days)
LIBERO-RIG is a VLA model benchmark test whose design perturbs only the robot hardware and setup, keeping the LIBERO tasks themselves completely unchanged.
LIBERO-RIG covers 11 perturbation dimensions, with 5 severity levels set for each dimension, combining into 210 test conditions across four LIBERO suites.
LIBERO-RIG's perturbation axes are divided into the perception level (camera mounting position, optical parameters, calibration offset) and the control and execution level (actuation gains, control frequency, proprioceptive noise, base pose, table height, gripper pad state).
LIBERO-RIG evaluates each condition using 500 episodes, covering nine recent VLA models, supplemented by rigorous statistical tests.
In the LIBERO-RIG evaluation, the aggregate success rate of VLA models dropped from a baseline of 0.932 to 0.767 at moderate severity, and further fell to 0.509 at the highest severity.
Moderate hardware perturbations alone can cost VLA models about 18 percentage points in success rate.
The magnitude of performance degradation varies enormously across different perturbation axes; some hardware changes barely affect performance, while others cause catastrophic drops.
The choice of action interface significantly affects the degradation patterns of VLA models under hardware perturbations, and different models exhibit markedly different failure characteristics when facing the same hardware perturbation.
Training on visual perturbations (such as visual augmentation techniques like random cropping, color jitter, and Gaussian blur) does not mitigate those hardware failures that do not alter the camera image.
The LIBERO-RIG release comes with a complete perturbation parameter manifest, ensuring that the hardware conditions of each episode are reproducible.
1 more timeline events
All Facts (11)
Feasible paths for hardware-aware training and adaptation mechanisms include: introducing proprioceptive signals such as joint feedback, torque readings, and control frequency as additional input channels into the policy network; performing domain randomization over control parameters, calibration offsets, etc. during training; and introducing online system identification during deployment.
50%UnverifiedLIBERO-RIG covers 11 perturbation dimensions, with 5 severity levels set for each dimension, combining into 210 test conditions across four LIBERO suites.
50%UnverifiedLIBERO-RIG's perturbation axes are divided into the perception level (camera mounting position, optical parameters, calibration offset) and the control and execution level (actuation gains, control frequency, proprioceptive noise, base pose, table height, gripper pad state).
50%UnverifiedLIBERO-RIG evaluates each condition using 500 episodes, covering nine recent VLA models, supplemented by rigorous statistical tests.
50%UnverifiedIn the LIBERO-RIG evaluation, the aggregate success rate of VLA models dropped from a baseline of 0.932 to 0.767 at moderate severity, and further fell to 0.509 at the highest severity.
50%UnverifiedModerate hardware perturbations alone can cost VLA models about 18 percentage points in success rate.
50%UnverifiedThe magnitude of performance degradation varies enormously across different perturbation axes; some hardware changes barely affect performance, while others cause catastrophic drops.
50%UnverifiedThe choice of action interface significantly affects the degradation patterns of VLA models under hardware perturbations, and different models exhibit markedly different failure characteristics when facing the same hardware perturbation.
50%UnverifiedTraining on visual perturbations (such as visual augmentation techniques like random cropping, color jitter, and Gaussian blur) does not mitigate those hardware failures that do not alter the camera image.
50%UnverifiedLIBERO-RIG is a VLA model benchmark test whose design perturbs only the robot hardware and setup, keeping the LIBERO tasks themselves completely unchanged.
50%UnverifiedThe LIBERO-RIG release comes with a complete perturbation parameter manifest, ensuring that the hardware conditions of each episode are reproducible.
50%