VLA Policy Fragility: The Selection–Completion Gap Under Synonym Rewriting

Synonym rewriting makes VLA pick the right object but fail the task, with fragility on both sides of the connector.
Studying GR00T N1.7 on LIBERO-Spatial, researchers find that synonym-substituted landmark words let the policy select the correct object but drop closed-loop completion from 100% to 67.6%. Interventions around the connector show semantics reach selection but not completion, with fragility on both sides. Fine-tuning the connector on diverse words repairs completion; open-loop scoring misses this. They recommend closed-loop evaluation with separate selection and completion scores.
Vision-Language-Action (VLA) models show promise in robot control, but a study of GR00T N1.7 reveals a key issue: when keywords in instructions are replaced with synonyms, the policy may successfully select the target object yet fail to complete the task. This challenges evaluating robustness by success rate alone and indicates that interventions must occur on both sides of the connector.

Synonyms Expose VLA Fragility
The study focuses on two LIBERO-Spatial tasks that differ only in their landmark word—the word telling the robot which of two bowls to pick up. When a synonym not seen in the training instructions is used, the policy moves toward the bowl indicated by the synonym rather than the one indicated by the original word, unlike with entirely fabricated words. However, in closed-loop episodes on one of the tasks, the synonym achieves only 67.6% completion, while the original word reaches 100%. This shows the policy understands the synonym's reference but fails at the execution stage.
LIBERO-Spatial is a set of robotic manipulation tasks in the LIBERO benchmark, specifically designed to probe a policy's understanding of spatial and referential relations. In this benchmark, each task is typically paired with a natural language instruction containing a landmark word that distinguishes visually identical objects in different locations, such as "left" in "put the bowl in the left drawer." LIBERO-Spatial tasks often differ in only a few words, making them well suited for controlled lexical substitution experiments: by replacing only the landmark word, one can observe whether the policy relies on lexical semantics or visual cues. Researchers often use closed-loop success rate as the primary metric, but the benchmark itself does not distinguish between the two failure modes of "choosing the wrong object" and "choosing the right object but failing to complete the action," which is the important context behind this paper's proposal of separate scoring.
Separating Selection from Completion
Success rate cannot reveal whether the policy chose the wrong object or failed to complete the task. By analyzing representations before and after the connector (the module between the vision-language model and the action generator), the researchers found: before the connector, replacing only the landmark word's vector with another landmark word's vector (pasting) causes nearly every episode to turn toward the other bowl; after the connector, pasting the landmark alone, the remaining instruction, or the image words rarely changes selection, but pasting any two simultaneously affects selection. Moreover, after the connector, hiding the landmark word still reduces the completion rate for those poorly performing synonyms. This proves that meaning reaches object selection but not the completion stage, and that fragility is distributed on both sides of the connector.
Connector in VLA architectures typically refers to the interface module between the vision-language model (VLM) and the action generator, responsible for converting joint language and visual representations into inputs the action generator can consume, such as a set of conditioning vectors or tokens. GR00T N1.7 adopts a similar "VLM backbone + action expert" structure, where the connector can be either a lightweight projection layer or a cross-attention adapter. Because the connector is the bottleneck through which semantic information enters the action space, researchers can determine at which stage semantic information is lost or misused by performing representation interventions (such as replacing or masking specific word vectors) before and after the connector. This intervention method draws on the idea of causal tracing, but here it acts on the closed-loop execution process of a robot policy rather than on open-loop prediction alone.
Repair Strategies: Fine-Tuning and Connector Copying
Fine-tuning on diverse landmark words can repair the task's completion rate for unseen synonyms, whereas fine-tuning on the original instructions is ineffective. Training only the connector on diverse vocabulary is sufficient, and even copying the connector of a policy fully fine-tuned on diverse words into a policy fine-tuned on the original instructions achieves the effect. Interestingly, open-loop scoring using only the initial frame misses the fact that weights outside the connector also carry part of the repair effect.
Open-loop scoring means giving only the initial frame or a fixed observation sequence, letting the policy predict an action sequence and computing its deviation from expert actions, without letting the policy actually execute in the environment. Closed-loop scoring, by contrast, lets the policy make decisions continuously based on observation changes caused by its own actions until the task ends. The two differ significantly in VLA evaluation: open-loop scoring cannot capture error accumulation, state distribution shift, or policy failures at critical steps, and thus may overestimate or underestimate robustness. This paper finds that open-loop scoring using only the initial frame misses the contribution of weights outside the connector to repair, because the influence of those weights only emerges during multi-step closed-loop execution. This also suggests that when evaluating robot policies, closed-loop metrics should be preferred, and when necessary, selection correctness and task completion should be counted separately.
Evaluation Recommendations: Closed-Loop and Separate Scoring
Based on this evidence, the researchers recommend: robustness to rewrites should be judged in closed loop (i.e., executing actions), interventions should occur on both sides of the connector, and selection and completion should be scored separately. This provides a more fine-grained perspective for evaluating and debugging VLA models.
One-line Summary
Synonym rewriting makes VLA choose the right object but fail the task, with fragility distributed on both sides of the connector.
Paragraph Summary
Studying GR00T N1.7 on LIBERO-Spatial tasks, the research finds that when landmark words in instructions are replaced with synonyms, the policy can correctly select the object indicated by the synonym yet fails to complete the task, with closed-loop completion dropping from 100% to 67.6%. Through representation interventions before and after the connector, the authors show that semantic information reaches the object selection stage but not the completion stage, and that fragility is distributed on both sides of the vision-language model and action generator. Repair experiments show that fine-tuning on diverse landmark words (especially training only the connector) restores completion, while open-loop scoring alone misses the role of weights outside the connector. They therefore recommend: robustness should be evaluated in closed loop, interventions should act on both sides of the connector, and selection and completion should be scored separately.
Related articles

Gemini 4 Neon Leak: A Potential Strong Contender Against Opus 5.5
A Hacker News thread suggests Gemini 4 Neon looks like an Opus 5.5 competitor. This article unpacks the product positioning behind this community signal, shifts in the top-model competitive landscape, and what it means for developers' model selection strategy.

Claude Fable 5.2's Self-Authored Short Film 'Fable' Sparks Discussion
A short film 'Fable,' credited to Claude Fable 5.2 for script, visuals, and directing, appears on Hacker News. This article analyzes the significance of full-pipeline AI creation, community reactions, and what it means for content creators.

Anthropic IPO Pricing: A Signal of AI Giants Going Public
Anthropic's IPO pricing draws attention. This article analyzes its potential impact on AI industry valuation, commercialization paths, and the competitive landscape, and highlights public-market expectations and challenges for AI companies.