Behind the RoboDojo Boom:
Revisiting Embodied Evaluation of Robot-Use Agents

TL;DR

Since the start of 2026, a large number of Robot-Use Agent (RUA) works have set new records on embodied leaderboards such as RoboDojo, and embodied intelligence appears to be advancing quickly. On closer inspection, a substantial share of these scores comes from three loopholes in current evaluation protocols. One concerns the budget. The other two are privileged information leaks, giving the agent information that the simulator has and a deployed robot does not:

• Unfair Budgets. Some works use execution budgets far above the defaults. With the step limit restored to the default, the success rate of RoboDawn's public runs drops from 47.1% to 1.0%.
• Accessible Verifier. Embodied benchmarks were built for policy models, which are not asked to decide when they are done, so an episode ends as soon as the simulator's success check passes. Many agent harnesses have carried this design over, and with an agent in control it becomes a privileged information leak: the success check tells the agent the task is done, in a tool return or by ending the episode for it, while a deployed robot has to judge that from its own sensors. When the agent decides on its own when to end the episode, RoboICL's mean score falls from 83.2 to 63.5 and RoboProbe's from 42.0 to 26.1.
• Overfitted Skills. Some works develop their skills and tools on the test tasks themselves, with the benchmark's verifier labeling the trials, and then report scores on the same tasks. Skills built this way generalize poorly: when the skills HexaShell evolved for one task are applied to four other tasks, they give either no improvement or a limited one.

Based on these findings, we propose the Revisiting RUA evaluation protocol: standardize and disclose execution budgets, give the agent only the information a real robot can also obtain, and do not compare results that use task-specific context with zero-shot results.

1. Behind the leaderboard boom

In recent years, embodied foundation models such as VLAs (and, more recently, world action models, or WAMs) have been the main subject of embodied evaluation. Representative works include RT-2RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (Brohan et al., 2023), OpenVLAOpenVLA: An Open-Source Vision-Language-Action Model (Kim et al., 2024), π0π0: A Vision-Language-Action Flow Model for General Robot Control (Black et al., 2024), and π0.5π0.5: a Vision-Language-Action Model with Open-World Generalization (Physical Intelligence, 2025). A second line of work is driven by large models. Early systems such as SayCanDo As I Can, Not As I Say: Grounding Language in Robotic Affordances (Ahn et al., 2022) and Code as PoliciesCode as Policies: Language Model Programs for Embodied Control (Liang et al., 2022) used language models to select predefined skills or generate control code. Since 2026, a group of systems has let large models observe the scene, call tools, write code, or directly control robot arms in a closed loop, for example RoboDawnTransferring the Intelligence of VLMs to Robotic Control (Guo et al., 2026), GPT-PolicyIn-Context Robot Learning with VLM Agents (Cheng et al., 2026), RoboICLRoboICL: Embodied In-Context Learning with GPT-6 Astra (Liu et al., 2026), HexaShellHexaShell: A Thin Harness for Exploring the Limits of VLM Agents as Robot Coding Agents (Gao et al., 2026), PyRUA-LeanFewer Tokens, Better Action: GPT-6 Astra Robot Agents with 14% Higher Success Rate but 65% Fewer Tokens (Si et al., 2026), EmbodiedEvoEmbodiedEvo: Self-Evolving Coding Agent For Physical Intelligence (Li et al., 2026), and VIAVIA: Visual Interface Agent for Robot Control (Hu et al., 2026). We call these systems Robot-Use Agents. They keep setting new records on simulation benchmarks such as RoboDojoRoboDojo benchmark and leaderboard (project website), LIBERO-PROLIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization (Zhou et al., 2025), and RoboTwin 2.0RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation (Chen et al., 2025).

How much of this rise in scores comes from progress in capability? Two numbers on a leaderboard are comparable only when they are measured under the same conditions, and most settings in embodied benchmarks were designed around policy models. When an agent controls the robot, we find three loopholes in current protocols that inflate scores:

Our starting point is simple. Simulation evaluation should give the agent only the information a real robot can also obtain: proprioception, the robot description file, camera RGB images with their intrinsics, and the task instruction. Everything else the simulator knows is privileged information, such as ground-truth poses, noise-free rendered depth, object models, and the verifier's judgment of the task. Robot learning already has a rule for such information: a policy may use it during training, as in Learning by CheatingLearning by Cheating (Chen et al., 2019) and RMARMA: Rapid Motor Adaptation for Legged Robots (Kumar et al., 2021), where policies read simulator ground truth while they learn, but it must work without it at deployment. Evaluation plays the role of deployment, so it should follow the same rule and run within a budget comparable to real deployment. The first loophole, Unfair Budgets, breaks the budget condition. The other two break the information rule, so we call them privileged information leaks: Accessible Verifier leaks task progress during execution, through the end-on-success design inherited from policy evaluation or tool response containing verifier information, and Overfitted Skills leak task solutions before evaluation. Many of these settings come from benchmark defaults or from conveniences added during development. We do not discuss motives and only measure their effect on scores.

2. Unfair Budgets: the same leaderboard, budgets more than ten times apart

RoboDojo sets a native control-step limit for each task at 25 Hz, for example 200 steps for general_pickup and 500 steps for play_Xylophone. These limits suit policies that output one action or a short action chunk per step, such as ACTLearning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (Zhao et al., 2023). An agent harness can replace this limit with "at most N decisions" or "at most N seconds". Each work makes this replacement differently, so scores on the same leaderboard come from different budgets. AI Agents That MatterAI Agents That Matter (Kapoor et al., 2024) made a similar point about language agents.

Embodied First ExamEmbodied First Exam (Embodied First Exam Team, 2026) (EFE) does not keep RoboDojo's per-task step limits (200 to 1,900 actions). It sets only a technical cap of 15,000 actions, and the binding constraint is a one-hour wall-clock limit per task. Recounted against the native limits, all 5 successes in standard mode exceed the limit, and in privileged mode only 2 of 16 successes fall within it (Figure 2a).

RoboDawn's released code uses _lift_step_limit() to set step_lim to 10**9, and episodes are instead bounded by 240 decisions and 9,000 s; the authors' public runs stop at 10 times the native limit. We reconstructed the control-step counts of the 210 episodes the authors released from their per-turn timelines: of the 99 successes, 97 occur after the native limit (Figure 2b).

Both works do set limits; they move the constraint away from control steps. The table below lists the limits each one actually sets, with RoboDojo's native settings for reference:

These limits differ in kind and in size, and none of them converts into the native step limits used for policy models. Some agent methods may genuinely need more test-time compute. Still, for deployment efficiency the step budget should not be relaxed without bound, and for fair comparison different works should use the same limit.

How evaluation should work

• Report the limits on control steps, decisions, and wall-clock time explicitly. Relaxing these limits is acceptable as a research setting, but it should be stated and compared separately.

3. Accessible Verifier: the simulator judges completion for the agent

A deployed robot can judge whether its task is complete only from its own sensors, and it has to decide for itself when to stop. The simulator always knows the answer: it reads the ground-truth state of every object and checks for success at every step. That judgment exists only in simulation, which makes it privileged information. The scoring rubric itself can be public, since a user would also tell a robot what counts as done. This section concerns the verdict: the verifier's live judgment of the current state, computed from simulator ground truth. Existing harnesses hand this verdict to the agent in two ways:

• Tool returns carry the verdict. RoboDawnTransferring the Intelligence of VLMs to Robotic Control (Guo et al., 2026) answers a done call with "NOT finished: the benchmark checker has not registered success"; GPT-PolicyIn-Context Robot Learning with VLM Agents (Cheng et al., 2026) returns native_episode_still_running: true; HexaShellHexaShell: A Thin Harness for Exploring the Limits of VLM Agents as Robot Coding Agents (Gao et al., 2026) and VIAVIA: Visual Interface Agent for Robot Control (Hu et al., 2026) return an episode_over field and a "TASK SUCCEEDED" message.
• The verdict ends the episode. This is the policy evaluation loop carried over unchanged. RoboICLRoboICL: Embodied In-Context Learning with GPT-6 Astra (Liu et al., 2026), RoboProbeAn Unexpected Robot Policy: Early Evaluations of GPT-6 Astra on RoboDojo and Beyond (Zhang et al., 2026), and WorldScape-R²SI have no done tool; RoboDawnTransferring the Intelligence of VLMs to Robotic Control (Guo et al., 2026) and GPT-PolicyIn-Context Robot Learning with VLM Agents (Cheng et al., 2026) provide one, and the episode continues after it is called. In both cases the episode ends when the verifier registers success or the budget runs out. The resulting score measures how well the agent executes the task; whether the agent can tell that it has finished is left out.

To measure the impact of this leakage, we modified RoboICL and RoboProbe: both fall into the second case (RoboICL's tool receipts themselves carry no success information), so we added a done tool and let the agent end the episode by calling it. RoboICL's mean score over 50 episodes on 10 tasks falls from 83.2 to 63.5, and RoboProbe's falls from 42.0 to 26.1 (Figure 3a). In 13 of RoboICL's 50 episodes, the agent called done before the task was complete.

Judging completion is hard, and it is a capability in its own right. RoboICL declared the task complete while the scissors were still on the table (Figure 3c). RoboProbe reported that all three bowls were stacked and stopped, yet the verifier counted only two (Figure 3d). Under the inherited end-on-success protocol, the agent never has to make such judgments, because the episode simply ends once success is detected. A deployed agent, by contrast, must make them in every episode, and the performance gap between the two harnesses quantifies what this costs today.

How evaluation should work

• Tool returns contain no information about whether the task is complete or how far it has progressed, and no live score is shown.
• The agent ends the episode by calling done or give_up, and the first call is final.

4. Overfitted Skills: test-task solutions are built into the agent

Systems can accumulate experience without changing model weights. VoyagerVoyager: An Open-Ended Embodied Agent with Large Language Models (Wang et al., 2023) keeps expanding a skill library in Minecraft, and ReflexionReflexion: Language Agents with Verbal Reinforcement Learning (Shinn et al., 2023) retries the same task with a memory of earlier failures. Some robot agent works apply this idea directly to RoboDojo's test tasks: they run the test tasks repeatedly, revise tools and skills according to the outcomes, freeze them, and then evaluate on other layouts of the same tasks. Examples include HexaShellHexaShell: A Thin Harness for Exploring the Limits of VLM Agents as Robot Coding Agents (Gao et al., 2026), EmbodiedEvoEmbodiedEvo: Self-Evolving Coding Agent For Physical Intelligence (Li et al., 2026), and RAXScaling Online Exploration for Robot Agents (Huang et al., 2026). OpenETAETA: A New Agentic Paradigm for Embodied Tasks (Chen et al., 2026) instead iterates across versions on the same test tasks: people help diagnose failures and revise the task wording, every attempt runs on the same seed, and the reported number is a Pass@3 accumulated across versions. These systems enter evaluation with privileged information about the test tasks: solutions found by trial and error on those tasks, usually with the benchmark's own verifier deciding which trials succeeded, and sometimes details of the scenes themselves. HexaShell's authors audited their tools after development and found that the tools for seven tasks held fixed object geometry, such as part sizes and templates of fixed scene parts; they replaced these constants with measurements made during the episode. A deployed robot usually meets tasks it has not practiced on, and no verifier labels its attempts. The resulting scores mainly reflect adaptation to specific tasks. The hardest part of embodied intelligence is handling tasks the system has never seen, and these scores do not measure that.

To measure how far such gains carry beyond the task they were developed on, we use the tools and experience notes (hereafter, skills) that HexaShell's coding agent accumulated for each task, which HexaShell has released. We freeze the skills from task A (cover_blocks), apply them to four other tasks, and compare against a baseline with only generic tools and a harness with each task's own skills. Each setting runs on 5 layouts with seed 0 (layouts 10–14) (Figure 4a).

Task A's skill lifts task A from 0 to 80. On the four other tasks, B and D stay at the baseline, and only C and E improve (20 → 60 and 28 → 68), by about half of what each task's own skill adds. Most of the gain stays on the task the skill was developed on, so the skill mainly records that task's solution. Scores obtained this way should not be compared side by side with zero-shot scores.

How evaluation should work

• Zero-shot experiments use only information provided by the benchmark, with nothing obtained from trial runs or development on the test tasks or test scenes.
• Works that use task-specific context should also report performance on new tasks that were not used during development.
• Works that develop on the test tasks should state what the development loop could access, such as the verifier's success judgments or numbers from the scene assets.

5. The Revisiting RUA protocol

• Rule 1 (Resource budget): Each work reports its limits on control steps, wall-clock time, and agent interaction turns. When a benchmark compares different works, these budgets should match.
• Rule 2 (Privileged information during execution): The agent receives only information a real robot can also obtain: proprioception, the robot description file, camera RGB images with intrinsics, and the task instruction. Tool returns do not reveal task success or progress, and the agent decides when to end the episode. Results that use privileged information should say so and be reported separately.
• Rule 3 (Privileged information from development): Skills, memories, and prompts accumulated on the target tasks through trial runs, development, or self-evolution are collectively called task-specific context. Such context carries privileged information about the test tasks, so results that use it should not be compared with zero-shot results, and such works should also report performance on new tasks.

6. Conclusion

Not all of the rise in leaderboard scores comes from progress in capability. Unfair Budgets, Accessible Verifier, and Overfitted Skills are three loopholes, and each can inflate scores while system capability stays the same. We call on benchmark maintainers to implement the Revisiting RUA protocol as a shared evaluation interface, and on RUA works to report their scores under it. A real robot has no automatic verifier and seldom practices on the exact task it is given, and the world does not pause while the model thinksSpotter: Let the Embodied Model Lead, and the VLM Reflect for It (Li et al., 2026). The closer evaluation protocols come to these constraints, the more likely it is that progress on simulation leaderboards will transfer to the real world.

Citation

@article{openrua2026revisiting,
  title={Behind the RoboDojo Boom: Revisiting Embodied Evaluation of Robot-Use Agents},
  author={{OpenRUA Team}},
  journal={URL https://openrua.ai/revisiting-rua},
  year={2026}
}

References

• Ahn et al. (2022). Do As I Can, Not As I Say: Grounding Language in Robotic Affordances.
• Black et al. (2024). π0: A Vision-Language-Action Flow Model for General Robot Control.
• Brohan et al. (2023). RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control.
• Chen et al. (2019). Learning by Cheating.
• Chen et al. (2025). RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation.
• Chen et al. (2026). ETA: A New Agentic Paradigm for Embodied Tasks.
• Cheng et al. (2026). In-Context Robot Learning with VLM Agents.
• Embodied First Exam Team (2026). Embodied First Exam.
• Gao et al. (2026). HexaShell: A Thin Harness for Exploring the Limits of VLM Agents as Robot Coding Agents.
• Guo et al. (2026). Transferring the Intelligence of VLMs to Robotic Control.
• Hu et al. (2026). VIA: Visual Interface Agent for Robot Control.
• Huang et al. (2026). Scaling Online Exploration for Robot Agents.
• Kapoor et al. (2024). AI Agents That Matter.
• Kim et al. (2024). OpenVLA: An Open-Source Vision-Language-Action Model.
• Kumar et al. (2021). RMA: Rapid Motor Adaptation for Legged Robots.
• Li et al. (2026). EmbodiedEvo: Self-Evolving Coding Agent For Physical Intelligence.
• Li et al. (2026). Spotter: Let the Embodied Model Lead, and the VLM Reflect for It.
• Liang et al. (2022). Code as Policies: Language Model Programs for Embodied Control.
• Liu et al. (2026). RoboICL: Embodied In-Context Learning with GPT-6 Astra.
• Physical Intelligence (2025). π0.5: a Vision-Language-Action Model with Open-World Generalization.
• RoboDojo benchmark and leaderboard. Project website.
• Shinn et al. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning.
• Si et al. (2026). Fewer Tokens, Better Action: GPT-6 Astra Robot Agents with 14% Higher Success Rate but 65% Fewer Tokens.
• Wang et al. (2023). Voyager: An Open-Ended Embodied Agent with Large Language Models.
• Zhang et al. (2026). An Unexpected Robot Policy: Early Evaluations of GPT-6 Astra on RoboDojo and Beyond.
• Zhao et al. (2023). Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware.
• Zhou et al. (2025). LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization.