How Should Demonstration Quality Be Used in VLA Pretraining? AXIS-Bench Provides Empirical Answers

Demonstration Quality Isn't About Quantity: The Core Tension in VLA Pretraining

One of the hottest directions in robotics in recent years is to pretrain Vision-Language-Action (VLA) models as general-purpose manipulation policies, then fine-tune them for specific tasks. But a long-avoided question is now surfacing: human teleoperation demonstrations vary widely in quality—so how exactly should this "suboptimal data" be used?

A new paper, What Matters in Quality-Aware VLA Pretraining: A Benchmark and Empirical Study, tackles this head-on. Using 5,000 naturally suboptimal human demonstrations and 24 target tasks, the authors systematically compare three quality-utilization strategies—weighting, filtering, and conditioning—across different reward definitions and label granularities.

The experimental controls are quite rigorous: demonstration counts and training budgets are matched within the same VLA, ensuring that what's being compared is the strategy itself rather than differences in data volume.

VLA (Vision-Language-Action) models are a class of models that unify visual perception, natural language instructions, and robot action generation into a single policy network. A typical architecture is: a vision encoder processes camera frames, a language model backbone (such as LLaVA, PaliGemma, etc.) fuses text instructions, and an action decoding head outputs end-effector poses or joint angle sequences. The pretraining stage typically performs behavioral cloning on large-scale human teleoperation demonstrations, teaching the model the mapping from "what it sees, what it hears, to what action it should produce"—and the fine-tuning stage then adapts to small samples of the target task. The value of VLA lies in cross-task generalization: a pretrained policy can be assigned new tasks via natural language without training separately for each task. But this also means the quality distribution of pretraining data directly determines the upper bound of downstream capability—this is precisely the background against which this paper discusses the "suboptimal demonstration" problem.

Three-Strategy Comparison: Conditioning Wins, But Details Have Reversals

The core findings can be summarized in a few points.

Conditioning is overall optimal. Across three VLA models, "approach-aware episode conditioning" consistently outperforms naive behavioral cloning in average success rate. Approach-aware means not only labeling whether a demonstration is "good" or "bad," but also encoding how it is good—the reasonableness of the execution path is incorporated into the conditioning signal.

The impact of label granularity varies by strategy. On average, conditioning prefers episode-level labels, while filtering prefers chunk-level labels. But this rule is not ironclad: the paper explicitly mentions "model-dependent reversals," meaning that with a different model, preferences may flip. This reminds practitioners not to blindly adopt conclusions and to validate on their own models.

Label strategy during fine-tuning also matters. One counterintuitive result: after conditioning-based pretraining, using constant-tag fine-tuning improves average success rate—and this improvement holds regardless of whether labels are used at inference time. This suggests that the value of quality labels lies not only in the training signal, but may also play some regularizing or distribution-alignment role during fine-tuning.

The paper additionally examines the impact of pretraining data scale, as well as the respective roles of quality labels during fine-tuning and inference, but the abstract does not provide specific numbers for this portion.

The core difference among the three quality-utilization strategies lies in how the signal of "demonstration quality" is fed to the model. Weighting multiplies the quality score onto the behavioral cloning loss, reducing the gradient contribution of low-quality demonstrations—essentially still a single strategy, just with different sample weights. Filtering sets a threshold and directly removes low-quality demonstrations, resulting in a smaller but cleaner training set; the problem is threshold selection and the sharp drop in data volume after filtering. Conditioning concatenates quality labels as additional input into the policy's observation, so that during training the model sees both the "quality description of the current demonstration" and the action, and at inference can optionally provide that label, thereby learning to adjust behavior based on quality signals. Conditioning does not change the loss function nor discard data, so it is typically more robust in high-noise, large-scale demonstration scenarios, at the cost of additional label engineering and input design.

AXIS-Bench: Making Quality Labels Recomputable and Trajectories Replayable

To support the above controlled comparison, the authors built AXIS-Bench. Several design choices of this benchmark are worth noting:

• Based on crowdsourced simulation demonstrations, providing recorded object states and contact states;
• Quality labels are recomputable: because states are fully recorded, researchers can recompute quality scores on the same trajectory using different reward definitions, without re-collecting data;
• Trajectories are replayable, supporting workspace-aligned rendering;
• Supports user-defined real-world target tasks, connecting the pipeline from simulation pretraining to real-robot deployment.

The value of this design lies in reproducibility. Research on quality-aware pretraining has previously been difficult to compare horizontally because data, labels, and budget were entangled; AXIS-Bench disentangles these variables.

Crowdsourced simulation demonstrations refer to recruiting ordinary people through crowdsourcing platforms to teleoperate a robotic arm in a simulation environment to complete tasks, rather than having experts collect data on real robots. This approach is low-cost and large-scale, but operator skill varies, naturally producing large amounts of suboptimal trajectories—which is exactly the real data distribution that quality-aware pretraining research aims to address. AXIS-Bench fully records the object states and contact states in each trajectory, meaning quality scores no longer depend on subjective ratings at collection time, but can be recomputed afterward with any reward function. Trajectory replay and workspace-aligned rendering further support fair cross-trajectory comparison: different operators' action spaces are mapped to the same coordinate system, allowing researchers to compare policy behavior under identical initial conditions. These designs decouple the three stages of "data collection—label generation—policy training," moving quality-aware research from case-by-case experiments toward a reproducible, controlled benchmark.

Real-Robot Experiment: Simulation Pretraining Raises Success Rate from 9.6% to 31.7%

The paper reports a real-world experiment result: using simulation pretraining alone, the success rate on the target task improved from 9.6% to 31.7%—more than a threefold increase.

The significance of this number is not in how high the absolute value is (31.7% is still far from practical), but in that it quantifies the gain magnitude of the "simulation pretraining + real-robot fine-tuning" route under quality-aware conditions. For teams building VLA data pipelines, this provides a reference baseline.

You may not have noticed that the 9.6% starting point means the policy is nearly unusable without pretraining, while 31.7% indicates that quality-aware simulation pretraining does bring substantial behavior transfer, although there is still a clear gap from reliable deployment.

Implications for Practitioners

If this work is distilled into actionable conclusions:

When choosing a quality-utilization strategy, conditioning is the default first choice, especially when you can provide episode-level quality or path annotations. Filtering and weighting are not ineffective, but they are more sensitive to label granularity, and their effectiveness depends more on the model itself.

Don't discard quality labels during fine-tuning. The simple operation of constant-tag fine-tuning alone can bring improvement at extremely low cost—worth trying.

Be cautious about transferring conclusions across models. The paper itself discovered model-dependent reversals, indicating that this methodology needs small-scale validation on your own model and data, rather than directly applying the paper's hyperparameters.

The project page is publicly available at https://axis-bench.github.io/; the specific availability of code and benchmark data needs further confirmation.

One-line Summary

Systematically compares weighting, filtering, and conditioning strategies for quality utilization in VLA pretraining; conditioning is optimal but model-dependent reversals exist.

Paragraph Summary

This article interprets the paper What Matters in Quality-Aware VLA Pretraining, using 5,000 suboptimal human demonstrations and 24 target tasks to compare weighting, filtering, and conditioning strategies for quality utilization under unified data and budget. The core conclusion is that approach-aware episode conditioning achieves the highest average success rate and prefers episode-level labels, while filtering prefers chunk-level labels, but model-dependent reversals exist and cannot be blindly applied. Using constant tags during fine-tuning also improves success rate. The authors release AXIS-Bench, supporting quality label recomputation and trajectory replay. In real-robot experiments, simulation pretraining raises success rate from 9.6% to 31.7%, demonstrating that quality-aware pretraining brings substantial transfer, though a gap from practical use remains.