Language Instruction Following Failures in Deployed VLAs

Language instruction following failures in deployed vision-language-action models arise from identifiable structural causes, including visual override of instruction semantics, unstable joint vision-language features, and training that excludes corrective experience. Research from 2025–2026 points to ways to help at each of these layers, but neither more data nor a single change to the architecture has yet solved the whole problem.

Why VLA language instruction following is harder than benchmark scores suggest

VLA models couple visual perception, natural language understanding, and action generation in a single policy. That coupling makes failure attribution difficult. When a robot selects the wrong action, the outcome alone does not reveal whether it misread the scene, lost the instruction’s meaning, or failed to translate that meaning into control. A systematic evaluation documents the resulting gap across a large suite of tasks, with models completing every training task without a single failure but generalizing to only about a fifth of held-out tasks.

Standard benchmark conditions make that fragility harder to see. Evaluations commonly pair short, templated instructions with familiar household objects, while real users phrase goals differently, refer to ambiguous objects, and change instructions mid-task. Those differences test more than manipulation skill. They test whether language actually governs action selection when the wording or goal departs from the demonstration distribution. A high completion score under familiar conditions does not show that a model can do this.

Visual override: how scene priors drown out instruction semantics

The most fundamental failure is visual override, in which a policy executes a visually plausible action even when the instruction contradicts the scene. The literature calls this behavior “linguistic blindness.” Evaluations of Pi0, Pi0.5, and OpenVLA-OFT show that the pattern extends across model families. Policies can complete familiar tasks even when instructions are logically impossible, showing that scene priors take over from instruction semantics in driving action generation.

The bread-and-plate example makes the distinction concrete. A command such as “put bread on plate” may fail even when the target is visually clear. Yet “pick up the brown round object and put it on the yellow plate” can succeed by redundantly naming perceptual properties. The task has not become semantically more demanding. Instead, the second wording supplies perceptual affordance signals that the policy can use more readily than the compact statement of intent.

Dataset composition helps explain why that shortcut is available. Most robot demonstration datasets pair everyday household objects with short, templated instructions, so the policy sees familiar visual arrangements over and over but gets little variety in the language. Its visual priors become well-calibrated while its language channel remains undertrained. ICBench isolates this imbalance by injecting controlled out-of-distribution instruction contradictions into environments built from LIBERO while holding the visual scene constant. This removes a key source of ambiguity. If the policy continues the familiar action despite a changed instruction, the failure cannot be explained simply by a changed visual environment.

Feature instability in joint vision-language encoding

Encoding changing visual observations together with text can cause systematic feature shifts that the downstream action policy cannot handle. The instruction may preserve its meaning while its encoded representation changes enough to disrupt action generation. Current VLAs show performance drops of up to 67.53% when a canonical instruction is merely reworded.

During execution under a paraphrased instruction, replacing only the language tokens in the final action block with tokens from a canonical run dramatically rescues success. The visual and state tokens remain unchanged. Because the intervention changes the encoded language pathway without changing those other inputs, it isolates a weakness at the junction between language features and the action-generation head. The result supports an architectural explanation rather than the simpler claim that the model cannot understand the paraphrase.

Instruction-action binding compounds that weakness in a different way. Analyses of fine-tuned π0.5 and GR00T-N1.7 identify policies that respond to both language and vision without combining them to select the action the task requires. Instructions cue familiar trajectory families, while visual feedback adjusts execution. A policy can seem to respond to language while using it mainly to call up a learned motion pattern, with little grounding of the requested goal in the current scene.

IndustrialVLA-Bench exposes the distinction between clean-task competence and robust language use. FastWAM and UnifoLM-VLA-0 differ by only a small margin on clean LIBERO tasks but diverge substantially on paraphrase variants. Clean performance is not a reliable proxy. The result shows that manipulation capability and language robustness are partially decoupled, so an evaluation that measures only the former can hide a weak encoding pathway.

Training regimes blind to failure

Success-only training reinforces both visual override and encoding instability by withholding the states in which those weaknesses become consequential. Demonstrations teach the policy how successful trajectories unfold within the training distribution. They provide little corrective experience when an unexpected or erroneous state breaks that pattern. Recovery is a separate learned capability. Without examples that ground reasoning in correction, a policy can retain strong visual priors yet have no learned behavior for returning from the distribution’s edges.

RoboSemanticBench, introduced at Findings of EMNLP 2026, shows why ordinary tests may miss this weakness in training. Many standard VLA evaluations can be solved through visual or instruction-action shortcuts that bypass complex instruction semantics. The benchmark tests whether post-trained models can use those semantics to pick and manipulate the right physical target. This changes what successful execution demonstrates. A familiar trajectory is no longer sufficient evidence of instruction following when the task requires semantic distinctions to determine the target.

VINE treats the exclusion of failure as a training choice. Teleoperation traces that contain failures are routinely discarded as noisy supervision, even though they record infeasible transitions and brittle behaviors. Those traces contain information that successful demonstrations omit. The framework treats negative experience as a structured learning signal, making failure relevant to the decision process.

Predictive monitoring research highlights that post-hoc feedback can arrive only after collisions or workspace conflicts have already occurred. Correction then follows an irreversible physical consequence. Success-only training and reactive supervision are therefore a problematic combination, because the first leaves recovery underdeveloped while the second waits for an error to become observable. The training objective has to be distinguished from the deployment requirement, which includes recognizing when a proposed continuation is likely to fail.

Why fixing the data alone will not fix the architecture

If the training set has narrow language templates and few corrective trajectories, a wider range of demonstrations should help the policy rely less on those patterns. More variety in language does make models somewhat more robust. The strongest case for data-scaling rests on the idea that the architecture seems brittle because the training data covers too few of the combinations of wording, scenes, and actions the policy must handle. On that account, richer supervision would teach the existing architecture to use its language channel more reliably.

GSR provides evidence that routing contributes to robustness by extracting instruction semantics independently through a frozen T5 encoder, then injecting those semantics into native visual and state representations, replacing unstable joint routing. A lightweight model using this approach rivals massively scaled baselines on LIBERO-Para without any paraphrase training data. The distinction is in the routing. This makes architecture a more efficient lever in that setting, because changing how meaning reaches the policy produces robustness that data scale alone does not establish.

That result has a clear boundary, however. GSR and many other grounding improvements are validated primarily on LIBERO-family benchmarks, which limits claims about their reach across deployment settings. The evidence supports addressing the translation from language features to action, but it does not establish that the same intervention will resolve every form of linguistic or physical variation. Data quality and diversity remain part of the explanation and the remedy.

Perception also causes failures that a change in language-routing cannot fix on its own. Errors in grounding, pose estimation, and depth reconstruction create systematic noise that propagates into policy learning. These are not merely alternative phrasings of one bottleneck. They concern the reliability of the representations on which embodied control depends. Bridging perception, language grounding, and control across heterogeneous platforms remains unresolved, so the case for architectural intervention must coexist with the need for better training evidence and more reliable perceptual inputs.

Architectural and data-level remedies now emerging in the literature

New ways to address visual override try to give the instruction more influence without retraining the core policy. IGAR, or Instruction-Guided Attention Recalibration, recalibrates attention toward instruction-relevant regions in a frozen VLA. VLA Grounder instead uses a “failure-derived command-space prior” to condition a pre-trained VLM that translates the user instruction and scene image into a short VLA-grounded command. The weights remain unchanged in both approaches. Their intervention points differ, with one adjusting attention and the other reshaping the command into a form the policy can act on more reliably.

For encoding instability, GSR changes how semantics enter the action pathway, while other work addresses failures when the task changes during execution. CoRe, or Counterfactual Realignment, from Case Western Reserve University research, handles mid-episode instruction changes and physical perturbations by using synthesized observations to imagine how to carry on from a recent viable state. It then minimally realigns the robot before returning control. No policy fine-tuning is required for this intervention. Its reported gains in success and fewer physical restorations show better execution recovery, but do not prove that the underlying joint encoding has become stable.

For training blindness, failure-aware systems make erroneous trajectories usable supervision. FailSafe automatically produces diverse failure cases paired with executable recovery actions. FailSafe-VLM, fine-tuned on that data, improves π0-FAST, OpenVLA, and OpenVLA-OFT on ManiSkill tasks without human supervision, with generalization across spatial configurations, camera viewpoints, and object and robot embodiments. RoboFAC takes a complementary route. Its failure-centric dataset combines erroneous manipulation trajectories and QA pairs across many scenes, supporting substantially higher failure analysis accuracy than GPT-4o and a meaningful improvement as an external supervisor in a real-world VLA control pipeline.

VINE uses negative experience to plan before execution begins. It separates high-level reasoning, called System 2, from low-level control, called System 1, and trains the reasoning layer on both successful and failed demonstrations. Candidate subgoals are scored by predicted success probability, allowing brittle branches to be pruned without altering the core policy. Failure becomes evidence for choosing the next step. This addresses a gap left by success-only learning, rather than asking the low-level controller alone to compensate after a poor decision.

Proactive monitoring complements these interventions by moving detection ahead of physical error. Predictive vision-language monitoring, evaluated in Frontiers in Robotics and AI in August 2026, forecasts execution-condition violations within a future time window and triggers fallback actions, outperforming reactive VLM baselines in navigation and manipulation settings. SAFE reads a VLA’s hidden states to produce a failure score that can transfer across tasks, which could let it detect failure from internal signals without waiting to observe the outcome. These approaches are not interchangeable or individually sufficient. Their results address different failure modes under different evaluations, and the available evidence does not establish their combined effectiveness or mutual validation across deployment settings.