Grounding Spatial Reasoning in VLA Policies

Vision-language-action models are now the default for general-purpose robot manipulation, and their aggregate success rates on standard benchmarks keep climbing. Yet the same policies that narrate scenes fluently and pick objects from familiar tabletop arrangements collapse when a camera shifts, a trajectory requires remembering where the end-effector has been, or an object must be grasped at a particular angle. The problem is not a lack of manipulation skill. It is that spatial grounding has never been measured or trained as its own faculty, and the failures it causes are structural, not incidental.

Why VLA policies fail at spatial tasks despite strong general performance

VLA policies inherit rich semantic priors from large-scale vision-language pretraining, and those priors show up in their behavior: they identify objects reliably, follow language instructions, and succeed often enough on aggregate benchmarks to suggest that generalized manipulation is close. But most evaluations have measured overall task success rather than isolating spatial grounding, which means a policy can succeed for reasons that have nothing to do with knowing where things are in three-dimensional space. The language grounding evidence audit makes exactly this point. System-level task success does not attribute performance gains to any specific spatial faculty, and a policy that passes the standard battery can still fail catastrophically when the spatial configuration changes.

Spa-Bench, a real-robot benchmark introduced in 2026, isolates spatial grounding from aggregate task success for the first time. Across three fully evaluated policies, success falls sharply from in-distribution to out-of-distribution spatial configurations. The crucial finding is what happens next: matched direct spatial instructions recover most of that lost performance. The manipulation capability is intact, the robot can physically do what the task requires, but the grounding that would tell it where and how to act is not. This is the signature of a structural problem inside the architecture rather than a general capability deficit. The robot can often do what is asked. It cannot reliably locate the action in space.

How a coordinate frame mismatch corrupts spatial predictions across viewpoints

The most direct source of spatial error sits in the relationship between what the model sees and what the model must predict. Vision encoders learn representations anchored in camera space, the coordinate system defined by the lens through which the robot observes the world. Action targets, however, are defined in robot base coordinates, the fixed reference frame at the robot's physical root. These two frames rarely coincide, and standard training never explicitly reconciles them. Vision-language pretraining works entirely in the image or camera coordinate system. When that backbone is adapted for control, its latent representations stay viewpoint-relative, but the supervision signal requires robot-relative predictions.

Robot datasets make the mismatch worse. Data is collected from diverse camera placements and hardware configurations, and the robot base is frequently outside the camera's field of view entirely. The consequence is a learning conflict: the same physical action seen from two angles produces two different image inputs, yet both are forced to share one robot-space supervision signal. The gradients that result pull the model in contradictory directions. This is not a problem that more data solves. OC-VLA argues that the camera-to-base ambiguity is particularly detrimental at large pretraining scale precisely because diverse camera viewpoints are common there. Scale amplifies the conflict rather than resolving it.

OC-VLA addresses the issue directly and with unusual economy. Using the camera's extrinsic calibration matrix, it transforms end-effector poses from robot base coordinates into camera coordinates before training, so that observation and action target share a single reference frame. The approach requires no substantial modifications to existing architectures and is plug-and-play compatible. SpatialVLA, from Shanghai AI Lab and accepted at RSS 2025, takes a complementary route. Its Ego3D Position Encoding injects three-dimensional spatial context from depth information into the visual input of a PaliGemma 2 backbone, informing the model of its own egocentric spatial geometry rather than relying on implicit frame inference. One approach aligns the frames. The other makes the frame perceptible. Both treat the mismatch as a design flaw to be corrected, not a nuisance to be tolerated.

Why reactive, frame-by-frame control produces temporal amnesia

Even a spatially corrected single-frame prediction is insufficient for tasks that unfold over time. The root problem is that mapping image inputs as current states to control actions ignores everything that happened before the current frame. Depth, position, and object configuration all change across a trajectory, and a purely reactive policy cannot distinguish the cup moved because the robot moved it from the cup moved because the scene changed. The history of the robot's own actions is information, and a policy that discards it will make decisions that are correct for the present instant but wrong for the trajectory as a whole.

TraceVLA, published at ICLR 2025, addresses this by encoding state-action trajectories as visual traces overlaid on the RGB image. The model gets a spatial-temporal record of past end-effector positions with no change to the architecture's input interface. Fine-tuned on a large set of robot manipulation trajectories, it outperforms OpenVLA by a substantial margin on SimplerEnv simulations and by a factor of more than three on real-robot WidowX tasks. An internal ablation within TraceVLA is especially revealing. Providing the same trajectory information as text tokens also improves performance over the baseline, but encoding it visually yields a substantially larger additional gain. The modality of spatial injection matters, not merely its presence.

Spatial Traces extends TraceVLA by projecting visual traces onto depth maps rather than RGB frames, so spatial and temporal information arrive together. On SimplerEnv, the mean number of tasks solved increases relative to both SpatialVLA and TraceVLA. The progression shows what temporal memory actually requires: not just a record of where the end-effector has been, but a record that lives in the same representational space as the geometry it describes. A reactive policy is amnesic by construction. Giving it a visual history restores the continuity that frame-by-frame control throws away.

How orientation blindness blocks the rotation half of 6-DoF manipulation

The subtlest and most recently formalized failure concerns rotation. Vision-language models are reasonably good at localizing objects in three-dimensional space, that is, position. They systematically fail to represent how those objects are oriented, that is, rotation. Six-degree-of-freedom manipulation requires both, and orientation blindness is therefore a ceiling on precision grasping and placement that coordinate-frame fixes and temporal memory cannot remove. A policy can know exactly where a mug sits on a table and still fail to grasp it if the handle faces the wrong way, because it never learned to represent the direction the handle points.

Traditional object pose estimation methods rely on pre-defined reference frames or CAD models, which limits generalization to novel objects and severs the link between geometric pose and the functional, task-relevant meaning a language model could otherwise use. A CAD model encodes how a specific object is shaped, not what grasping a particular instance of it requires in context. SoFar, a NeurIPS 2025 Spotlight paper, introduces semantic orientation, a language-grounded, reference-frame-free representation that describes object orientation in terms a VLM can ground semantically rather than geometrically. In execution tasks within Open6DOR V2, SoFar achieves an overall success rate more than double that of the next best baseline. The gain comes from teaching the model what orientation means in functional language, not from adding more geometric precision to an already strained representation.

SoFar also introduces Open6DOR V2, a large-scale benchmark for 6-DoF object rearrangement, and OrienText300K, a dataset of hundreds of thousands of diverse objects and orientations with eight million images. The data infrastructure for orientation-aware training is now available at scale, and its absence before explains in part why orientation was the last of the three gaps to be formally characterized. It is important to keep the orientation gap distinct from depth blindness. Depth helps recover the third positional dimension, but it does not recover roll, pitch, or yaw. A system with perfect depth can still be orientation-blind.

What happens when all three gaps compound: depth blindness as a shared substrate

Beneath all three failures lies a common deficit. Most VLA models ignore explicit depth information and treat the three-dimensional world through two-dimensional projections. Approximating depth from frame-to-frame comparison loses sensitivity to object size, camera distance, and fine movements in ways that compound coordinate errors, temporal confusion, and orientation mistakes simultaneously. Depth approximation from motion parallax degrades at the extremes of scale, very small or very distant objects, and is blind to static scenes where the robot has not yet moved. Temporal amnesia and coordinate mismatch errors are worst precisely when depth cues are weakest.

SpatialVLA's Ego3D Position Encoding attacks this at the input level by injecting depth from a dedicated depth channel. Its Adaptive Action Grids go further: they discretize continuous movements into spatial grids calibrated to the actual data distribution, reducing the action space to one that reflects real three-dimensional structure rather than uniform Cartesian bins. VLA-Talker, the framework within In-Context VLA, takes an agentic approach instead. Rather than enriching the static input, it delegates evidence acquisition to an external tool-use loop comprising an open-vocabulary detector, a monocular depth estimator, and a vision-language model fallback, queried on demand. The policy consumes grounded depth evidence rather than inferring it from the same static features that drive action prediction.

The compounding effect also explains why Spa-Bench's out-of-distribution performance drop is so large. Changing the spatial configuration simultaneously shifts the coordinate mapping, breaks trajectory priors, and changes the orientation context. All three gaps activate at once, and a policy that has never seen them interact has no fallback. Depth is the shared substrate because it underlies every one of the spatial faculties that fails. Patching coordinate alignment alone leaves temporal and orientation errors intact. Patching temporal memory alone leaves the coordinate conflict unresolved. The gaps are not independent, which is why partial fixes produce only partial improvement.

What current approaches actually fix and where they still fall short

Each published fix addresses one or two of the three gaps effectively but introduces a trade-off, whether latency, data requirements, or hallucination risk, that prevents any single approach from constituting a complete solution. OC-VLA resolves the coordinate mismatch and is plug-and-play, but it requires extrinsic calibration data that may not be available across diverse deployment hardware. TraceVLA and Spatial Traces address temporal amnesia through visual prompting, which is lightweight, but they depend on keypoint tracking quality. In cluttered scenes where keypoints are occluded, the trace degrades, and the memory it encodes degrades with it.

SoFar's semantic orientation addresses the rotation gap, but it is evaluated primarily on object rearrangement. Its generalization to contact-rich tasks such as cap screwing or liquid transfer, exactly the precision tasks flagged in LabVLA's laboratory manipulation setting, remains an open question. Chain-of-thought spatial reasoning, exemplified by CoT-VLA at CVPR 2025, achieves meaningful improvement over OpenVLA on LIBERO simulation and real-world manipulation, but it inherits hallucination from its pretrained multimodal LLM backbone. Generated plans can reference incorrect object attributes or spatial relations even when multi-step planning improves on average. In-Context VLA argues that the problem with generative chain-of-thought is structural. Reasoning and action tokens compete inside a single autoregressive loss, so the policy drifts toward fluent narration rather than accurate action. Consuming grounded language via tool-acquired evidence outperforms generating it. The latency and hallucination objections are symptoms of a conflicted objective, not fixable by better prompting alone.

Fine-tuning improves all of these systems, but it demands extensive data and prolonged training that is impractical for many real-world deployments. The gap between benchmark performance and deployment viability remains large. A navigation grounding study noted in the research adds a further caution: visual grounding primarily acts as a trajectory-length regularizer rather than improving per-unit-distance spatial reasoning, and the stop instruction failure mode remains unsolved. Grounding improvements measured on manipulation benchmarks may not transfer to navigation contexts. The honest assessment is that each approach narrows one gap while leaving the others partly open.

How the field is restructuring around these gaps rather than working around them

The most significant development in this area is not any single model but a shift in methodology. Gap-specific benchmarks, orientation-aware datasets, and architectures that treat three-dimensional grounding as a first-class design requirement are emerging, replacing the practice of patching an otherwise unchanged model. Spa-Bench separates spatial grounding performance from aggregate task success in a real-robot setting, and that methodological advance is likely to reframe how the community measures progress, because aggregate benchmarks have obscured the gaps documented here. Open6DOR V2 and OrienText300K provide the data infrastructure for orientation-aware training at scale, a prerequisite whose prior absence explains in part why orientation was the last of the three gaps to be formally characterized.

StrataVLA and ROCKET, both from 2026, are pushing toward multi-layer spatial alignment as a standard architectural component, StrataVLA via hierarchical geometry grounding and ROCKET via residual-oriented multi-layer representation alignment. The move reflects a field that no longer treats three-dimensional grounding as an optional add-on patched post-hoc by individual researchers. The laboratory manipulation domain, addressed by LabVLA from Zhejiang University, Shanghai AI Lab, and Harbin Institute of Technology, represents a natural stress test. Pipette aspiration, cap screwing, and liquid transfer demand fine spatial precision and reliable contact control in all three gap dimensions at once, so the setting is a productive testbed for measuring whether architectural grounding is genuine or still benchmark-specific.

The direction of travel runs from feature-level patches, a depth encoding added to an otherwise unchanged architecture, toward system-level redesign in which coordinate frame handling, temporal memory, and orientation representation are co-designed from the start. That is what genuine spatial grounding in robot policies actually requires: not a single fix to any one gap but a coherent design that resolves all three together. The three failures are interlocking because they share a common substrate in the absence of explicit three-dimensional reasoning, and the field's restructuring is an acknowledgment that working around them has reached its limit.