Reward Shaping With Vision-Language Models for Robot RL
How vision-language models can replace hand-built robot rewards.

Reward shaping with vision-language models uses visual and semantic knowledge learned at web scale in place of brittle robot RL rewards designed by hand. VLMs can guide progress throughout execution, but useful guidance requires reliable judgments, temporal context, and a shaping formulation that preserves the original task objective.
Why hand-engineered reward functions fail robotic RL at scale
Sparse terminal rewards show whether a robot succeeded in the end, but leave open which actions along the way helped. That gap grows with task length. When successful outcomes are rare, the agent receives little information about whether its behavior advanced the task, stalled it, or reversed earlier progress. Zhang et al.’s survey frames this as a credit-assignment failure, because useful behavior remains difficult to distinguish from irrelevant motion.
Hand-engineered dense rewards supply intermediate feedback, but their usefulness depends on task-specific assumptions. They are brittle across tasks and embodiments, so each new setting demands another engineering cycle. Human demonstrations and preferences offer another route to reward learning, but collecting them is expensive and difficult to scale. Even when policy improvement is automated, it still depends on people to design rewards. The alternative begins with a question about what changes when that bottleneck is replaced by a model that already encodes what task success looks like.
What VLMs bring to reward design that manual engineering cannot
VLMs interpret visual observations together with natural-language descriptions, which is precisely the pairing a visual reward function needs. Their image-text pretraining supplies knowledge that is not confined to one robot or task. A single model can, in principle, evaluate whether a drawer was opened, a cup stacked, or a ball placed in a basket from a text prompt. The goal becomes something to describe. This changes the entry cost of reward design without requiring a separate heuristic for every task.
A reward model must locate the current observation within an unfolding execution and infer which subgoals are complete. The same observation can point to different stages when the instructions change, so language conditioning helps define what counts as progress. Success detection alone cannot resolve that ambiguity. The survey follows how the field has moved from learned goal-proximity metrics through foundation-model similarity scores to language-conditioned progress rewards as it works to meet this need.
Three distinct mechanisms for turning VLM knowledge into a reward signal

Zero-shot scoring uses a pretrained VLM to judge the current visual state against a task description. No labeled robot data or fine-tuning is required. TOPReward asks a binary completion question and extracts the affirmative token’s probability as a continuous signal, avoiding the calibration problems of generated numeric estimates. RL-VLM-F, presented at ICML 2024, generates rewards from task descriptions and visual observations without manual crafting. This route offers broad applicability, but its reliability depends on judgments that generalist models can get substantially wrong.
Code generation instead uses the model to write an executable reward function. The output is inspectable and editable. EUREKA, developed by NVIDIA and UPenn in 2023, uses GPT-4 to read environment source code, generate Python rewards, and improve them through evolutionary search and reward reflection. It solved dexterous pen spinning and outperformed human experts on a large majority of tasks across diverse robot morphologies. Text2Reward and Code as Reward build on this approach, making debugging easier than with black-box scoring, though the generated code can still have discontinuities that an agent can exploit.
Specialist reward models instead learn robotic judgments from curated training data. Their precision comes with a data requirement. RoboReward, developed by Stanford and UC Berkeley in January 2026, trains billion-parameter models on 45,000 scored robot episodes across diverse embodiments, with calibrated negatives and near-misses produced through counterfactual relabeling and negative clipping. DenseReward, which came out in 2026, uses failure synthesis to train a dense reward model. These specialists outperform much larger generalist VLMs on robotic tasks, but require curated data and training effort beyond prompting alone.
Why static, single-step reward signals break down on long-horizon tasks
MVR, presented at ICLR 2026, documents how image-text similarity can reward a humanoid for freezing in a matching pose while failing to produce rhythmic motion. Occlusion introduces a different distortion, because a robot limb blocking the camera can make a successful grasp appear unsuccessful. On long-horizon tasks, a single goal-level query also gives nearly flat rewards early on because how much a state looks like the final state says little about progress along the way.
MVR uses video-text similarity from more than one viewpoint and a learned state-relevance function to give dense guidance early in the task. Its VLM has less influence as behavior improves, which reduces ongoing conflict with the task reward. Subtask decomposition offers another response to weak intermediate feedback. VLLR and RMTL divide long tasks into verifiable stages with separate prompts, making progress visible before terminal success. VLLR reports up to 56% absolute success rate gains over the pretrained policy on CHORES, while limiting inference expense by using VLM estimates to initialize the value function only during a brief warm-up.
Process reward models as the field's current answer to dense, reliable feedback
Process reward models judge how execution changes between steps rather than only how closely a frame resembles the goal. That distinction makes stagnation and regression observable. It also allows partial completion to receive feedback before the terminal outcome is known. Robo-Dopamine, presented at CVPR 2026, trains a cross-task general process reward model on multi-view video and connects its dense signals to RL through policy-invariant shaping. It improves a policy from near-zero to a high success rate within roughly 150 online rollouts, approximately one hour of real robot interaction.
Robo-Dopamine 2.0, introduced in August 2026, adds history-conditioned pairwise rewards and an OOD-aware signed progress space. History resolves ambiguity that appearances alone leave open. Hidden contact events, repeated phases, and reordered subgoals can produce visually similar states with different underlying progress, a problem known as temporal aliasing. The model reaches 86.8% mean success on RoboTwin in downstream RL. DenseReward takes a complementary route, using synthesized failure trajectories to learn per-timestep distinctions across failure modes.
Robometer, presented at RSS 2026, outperforms RoboReward on a specific benchmark by addressing inaccurate rewards and unreliable stage transitions. RoboRewardBench, RoboRMBench, and RoboProcessBench provide infrastructure for testing such differences systematically. The field can therefore assess reward quality more directly rather than relying only on downstream policy results.
The policy-preservation problem that naive reward shaping creates

An accurate VLM score can still produce an incorrect RL objective. Simply adding that score to the task reward may change which policy is optimal, allowing the agent to maximize auxiliary feedback at the expense of task completion. Better visual judgments do not repair an unsafe way of incorporating them into the objective.
Potential-based reward shaping, or PBRS, gives guidance along the way through differences in a potential function from one state to the next, while keeping the optimal policy set unchanged. Its traditional difficulty is constructing that potential. VLM-PBRS, introduced by L3S and Leibniz University Hannover in June 2026, learns it from VLM preferences over image pairs, replacing hand-engineered terms. The structural guarantee tolerates imperfect preference labels without changing the original optimal policies. This also lets policy learning use smaller, cheaper VLMs in place of frontier models.
MVR reduces VLM influence as behavior aligns with the target, allowing the task reward to take precedence. Robo-Dopamine uses a policy-invariant shaping mechanism to prevent its process signal from overriding the original objective. These mechanisms should not be treated as interchangeable guarantees. The shaping formulation needs deliberate selection before the rest of the reward design is complete.
Reward reliability: where even strong VLMs make gross errors
RoboReward reports a drawer-opening episode in which the robot misses the handle, the drawer stays closed, and the gripper falls over. Gemini Robotics-ER nevertheless assigns 5 out of 5. The judgment reverses failure and success completely. RoboRewardBench’s evaluation of 22 proprietary and open-source VLMs finds that no model excels across all tasks, showing why broad generalist capability is not a reliable substitute for manipulation-specific reward accuracy. Such errors can corrupt learning with noise that fails to average out.
RoboRMBench, presented at EMNLP 2026, isolates another failure mode by holding the trajectory fixed and changing only the instruction’s wording. Meaning-preserving paraphrases can substantially change the predicted reward. Under action-goal perspective shifts, some generalist models assign both failure-level and success-level predictions to the same trajectory for more than half of the benchmark. The benchmark treats this as a distinct weakness requiring targeted solutions even as general-purpose capability improves.
These findings explain the emphasis on specialist training. RoboReward’s calibrated negatives, DenseReward’s synthesized failures, and Robo-Dopamine’s process supervision address precision gaps left by general pretraining. The bottleneck has shifted from producing a reward to trusting it. Robot-specific training provides a concrete response, but not a settled account of what future models will require. Whether scale alone can close the generalist-specialist gap, or robot-specific fine-tuning remains necessary, is still unresolved.
Applying VLM reward shaping to VLA fine-tuning, not just RL from scratch
Refining a pretrained vision-language-action model with RL places greater demands on reward reliability than learning a policy from scratch. Models such as π₀ remain vulnerable to compounding errors, unseen scene changes, and off-trajectory states that require recovery. Demonstrations do not cover every state encountered during refinement. A useful reward model must judge behavior beyond the demonstration distribution as well as recognize successful sequences it has seen before. Robo-Dopamine 2.0 explicitly targets these out-of-distribution execution states and the temporal ambiguity that accompanies them.
Real-robot fine-tuning also tightens the interaction budget. Human-labeled rewards are expensive, and physical rollouts make inefficient exploration costly. This gives dense feedback practical significance beyond faster convergence in simulation, while VLLR’s warm-up-only strategy shows how inference expenditure can also be constrained. Neither efficiency nor specialization establishes parity with human judgment. RoboReward 8B narrows the gap to human-provided rewards but does not close it, leaving human rewards as the standard against which automated feedback is evaluated.
Practical design decisions when implementing VLM-based reward shaping
For implementation, the first decision is what judgment the model should make. Binary completion queries avoid free-form numeric calibration, while pairwise preferences ask the comparatively simpler question of which observation represents more progress. Whether noisy preferences are safe still depends on the formulation. When interpretability and reward-hacking audits are priorities, generated code offers an inspectable alternative to black-box scoring.
The next decision concerns how much execution context the reward needs. Short tasks may need only a success query. Long-horizon manipulation needs subtask feedback or per-step process rewards to keep signals from staying nearly flat, while repeated phases and hidden contact events call for history conditioning. Multi-view observations are appropriate when the task geometry makes a single camera vulnerable to occlusion. The shaping formulation must then constrain that feedback, using potential-based or policy-invariant shaping when policy preservation is required, or state-dependent decay to reduce persistent auxiliary influence.
Which of these designs can be used during training depends on cost. Smaller VLMs with PBRS, warm-up-only queries, and specialist reward models offer alternatives to invoking frontier models at every step. Specialists also offer a favorable cost-accuracy trade-off on robotic tasks. Before committing to a full training run, the remaining low-cost check is prompt stability. Hold the trajectory fixed, paraphrase the task description without changing its meaning, and test whether the reward remains consistent.
Sources
- Progress Reward Modeling for Robotic Learning: A Comprehensive Survey
Supplied the survey framing of credit assignment, the progression from goal-proximity metrics to language-conditioned progress rewards, and the VLLR results.
- Robo-Dopamine 2.0: History-Conditioned and OOD-Aware Process Reward Modeling for Robotic Manipulation
Described history-conditioned pairwise rewards, the OOD-aware signed progress space, temporal aliasing, and the RoboTwin result.
- RoboReward: General-Purpose Vision-Language Reward Models for Robotics
Provided RoboReward's training data, counterfactual relabeling, the drawer-opening failure example, and the benchmark evaluation of 22 VLMs.
- MVR: Multi-view Video Reward Shaping for Reinforcement Learning
Supplied the MVR failure modes of static similarity rewards, multi-view video rewards, and decaying VLM influence.
- Automating Potential-based Reward Shaping with Vision Language Model Guidance
Explained how VLM preferences over image pairs can build a potential function for policy-preserving shaping.
- RoboReward: General-Purpose Vision-Language Reward Models for Robotics
- DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation
Provided the DenseReward approach of synthesizing failure trajectories to train a dense reward model.
- Same Trajectory, Contradictory Rewards (ROBORMBENCH): Paraphrase Fragility in Vision Language Reward Models
Supplied the paraphrase-fragility findings, where changing only the instruction wording shifts predicted rewards.


