TL;DR

We introduce uni-robot-agent, a unified and modular harness for robot use without privileged inputs. Designed for generalizable agentic training. It unifies interaction records across environments and supports efficient experience collection for open models.

On RoboDojo1, Qwen3.8-Flash-Next2 reaches 31.39 Score and 25.00% success rate; GPT-6 Astra3 reaches 51.90 Score. On physical robots, Qwen achieves a zero-shot Score of 53.50, compared with 47.75 for fine-tuned π₀.₅4. With Astra, the harness reaches 80.50 zero-shot. These results support a reusable foundation for effective robot use with open models.

One interface, from reasoning to robot action
Agent
Tools
Environment adapters
Observations & execution receipts

Models call shared tools; environment adapters execute actions and return evidence for the next decision.

RoboDojo

Simulation · direct control · Score (0–100)

Qwen3.8-Flash-NextRoboProbe5 · zero-shot
1.90
+29.49 points
uni-robot-agentzero-shot · no privileged inputs
31.39
with depth · 35.89
GPT-6 Astra · publishedRoboProbe · zero-shot
28.97
010203040

Real-World Benchmark

Physical robots · 19 tasks · Five dimensions · Total (0–100)

Qwen3.8-Flash-Nextuni-robot-agent · zero-shot
53.50
RGB + D405 depth
π₀.₅fine-tuned · 10 hours/task
47.75
RGB only
Dream-16zero-shot
43.50
RGB only
0102030405060
Figure 1 | The paper’s overview, adapted for the blog. Shared tool semantics connect models to environments and return execution evidence. With the same Qwen model, changing the harness from RoboProbe to uni-robot-agent raises Score from 1.90 to 31.39 (+29.49 points) without privileged information. The dashed extension shows 35.89 with depth; the published Astra–RoboProbe result is a separate reference. In the real-world comparison, our agents use D405 sensor depth; π₀.₅ and Dream-1 use RGB only. The π₀.₅ baseline is fine-tuned with 10 hours of robot data per task. Published baselines7 retain their own protocols. Full results and settings appear in Evaluation.

1. Introduction

Frontier models such as GPT-6 Astra have shown surprising zero-shot robot manipulation capabilities, as demonstrated by RoboProbe. These robot-use agents bring general problem-solving abilities—task decomposition, planning, and failure recovery—to physical interaction. Such abilities are valuable for robotics, yet difficult to learn from limited robot data alone.

But proprietary models constrain how researchers can develop this approach. Without access to their weights, we cannot directly train them to become better robot-use agents. API dependence also makes large-scale interaction costly and limits model-level optimizations, such as efficient caching, even though inference speed is a major bottleneck.

Open-weight models offer a path forward, but often fall short in spatial understanding and agentic capabilities. One response is to supply simulator ground truth, oracle success feedback, or task-specific skills. Before attributing failures to the models themselves, however, we need to establish what they can already do with an effective, generalizable harness. That gives us a stronger foundation for deciding how future robot-specific training should improve them.

We built uni-robot-agent to provide that foundation: competitive robot use with open models today, and infrastructure for future training. Unified tool interfaces and feedback formats span models, robots, tasks, and environments, making simulation and real-world experience easier to combine for training. Efficiency makes large-scale interaction more practical; modularity lets researchers change models, tools, and robot backends as their methods evolve.

Our central finding is that well-structured information and reliable tools with informative feedback can be enough for effective robot use. With camera calibration and robot descriptions, Qwen3.8-Flash-Next reaches 31.39 Score on RoboDojo without privileged inputs, surpassing the published Astra–RoboProbe result of 28.97. Across 19 physical-robot tasks, Qwen reaches 53.50, outperforming zero-shot Dream-1 (43.50) and performing comparably to fine-tuned π₀.₅ (47.75). In this real-world comparison, our agents use sensor depth; the two baselines use RGB only.

With this foundation in place, we ask what makes robot use work. Comparisons across model generations, families, and sizes reveal four recurring bottlenecks: visual understanding, spatial grounding, geometric coding, and embodied reasoning. These point to complementary targets for future training. We then examine the effects of privileged information and skills, followed by the cost and speed of interaction.

2. A Unified and Modular Harness

2.1 Modular Architecture

A tool call passes through five layers. The agent layer manages model interaction and context; orchestration schedules and dispatches requests. Shared operations in the tool layer access environment, perception, and action-model capabilities through services, which manage runtime resources. Concrete simulator, robot, and model implementations sit in the backend layer. Results return through these interfaces to the agent, completing the loop.

System architecture

Request / operationState / receipt
RECEIVES

MOTION TOOLS

RETURNS

Figure 2 | System architecture. Five layers, following the paper’s architecture. Animated connections explain interface semantics; they are not a live execution trace.

2.2 Unified Interaction Interface

Shared motion, perception, skill, and episode tools use common arguments, units, and feedback. Environment descriptions identify available arms, cameras, and optional capabilities; adapters translate shared operations into native control. Adding an environment requires an adapter rather than reimplementing every tool, and the common interaction format lets us combine experience for subsequent training.

A shared interface is also a foundation for agentic training. AgentGym8 collects trajectories across diverse environments in a common interaction format and uses filtered demonstrations for supervised fine-tuning. Agent Lightning9 addresses a complementary layer: structured invocation records separate execution from training, with LLM inputs, outputs, and rewards extracted for policy optimization.

For robot-use agents, this motivates preserving the link between observations, tool calls, and feedback while keeping operation semantics consistent across environments. Such records make experience easier to combine and prepare for training open models. Data selection, learning objectives, and reward design remain separate questions; training on the collected experience is future work.

Why does a unified tool interface enable generalizable agentic training?

From an LLM’s perspective, search and coding tasks across different environments can be learned through the same set of tools. This shared interface can more effectively promote cross-environment generalization.

Open-weight agentReasons over observations
Unified tool interface
MotionPerceptionSkillsEpisode
Consistent arguments, units and feedback
Environments & embodiments
Task trajectoriesDifferent goals · shared tools
Sandbox ABlock stacking

Stack the red block on the blue block.

  1. Locate both blocks
  2. Grasp & align
  3. Stack & release
Sandbox BObject handover

Pass an object from the left arm to the right.

  1. Present the object
  2. Secure second grasp
  3. Transfer & release
Physical robotPeg insertion

Insert a peg while stabilizing the socket.

  1. Stabilize socket
  2. Align the peg
  3. Insert & verify
Agentic TrainingCross-environment generalization

One tool interface across environments and embodiments.

Figure 3 | Different goals and arm configurations, expressed through the same tool interface. Consistent interaction trajectories can be combined for agentic training across environments.

2.3 Efficient Execution

Waiting for every robot movement to finish also holds up reasoning that could proceed independently. A background tool call returns a job handle—a reference the agent can use to retrieve its result later—so the agent can continue reasoning while the tool runs. An immediate call waits for completion. Decisions that need a result still wait.

Tool execution follows resource constraints. Calls controlling the same robot arm enter a queue and run in submission order: a second movement waits for the first to finish, even as the agent prepares later steps. Independent operations, such as reading an existing snapshot, can overlap movement. This forms a two-stage pipeline between reasoning and execution. If new feedback changes the plan, flush_queue cancels calls that have not yet started for the specified resource, allowing the agent to submit revised actions.

Agent–tool pipelining

0.0 / 12illustrative time
The same model stages and tool operations begin together.
Figure 4 | Agent–tool pipelining. R = reasoning · T = tool execution. Block widths show duration; vertically aligned blocks run concurrently. T2 queues from 4 to 5 before entering execution. Timings are illustrative, not measured speedups.

2.4 Interaction Without Privileged Inputs

Depth measurements can vary or be missing even in a static physical scene, and simulator task-verifier feedback is unavailable on real robots. We therefore test whether agents can infer spatial relationships and assess progress from more limited observations.

Our main simulation setting provides RGB images, camera calibration, robot proprioception and descriptions, and task instructions. It excludes depth measurements, supplied object CAD assets, hidden scene annotations, task-verifier feedback, and external web access. These restrictions also apply to tool computations: localization must derive coordinates from permitted observations, such as through multi-view triangulation. Tools return measured robot poses and execution outcomes; task-verifier results are used only for evaluation.

Task instructions. Within these input constraints, agents still need a clear task specification. Some RoboDojo instructions omit or ambiguously describe task requirements. We add brief clarifications: each red button’s prescribed press count must be confirmed with the blue button, and tower building requires a specified arrangement of blocks and boards. These additions clarify task rules without providing scoring rubrics or verifier logic. We also manually check that models do not access prohibited information.

2.5 Key Design Choices

Physical interaction can leave the agent with stale images, uncertain action status, or incomplete evidence of success. These six choices keep its next decision tied to what was observed and what actually ran.

Physical consistency

Keep observations and execution records tied to the physical state.

Use the same snapshot
Bind RGB, calibration and robot state to one immutable snapshot.

If the robot moves after an image is captured, a newer camera pose can misinterpret the selected pixel. We bind images, calibration, and robot state to one immutable snapshot, so localization uses the state that produced the image.

Check before repeating
Reject duplicate requests and recover uncertain execution records.

A missing reply may hide an action that already ran. Request identities and execution records reject duplicates. When completion is uncertain, further actions are blocked while the session recovers the record; unresolved requests stay explicitly uncertain.

Action precision

Coordinate control and respond to measured progress.

Move and grip together
Coordinate arm motion with gripper opening.

Separate movement and gripper commands cannot adjust the opening during a movement. The move tool accepts a target pose and an optional gripper opening together, allowing the arm and gripper to act simultaneously.

Stop based on progress
Monitor motion progress and return control when stalled.

A fixed step count can stop too early or waste motion on an unreachable target. The tool generates intermediate targets from measured poses and checks progress after each step, stopping on arrival or returning control when motion stalls or reaches an execution limit.

Context management

Interpret outcomes and retain the evidence needed for the next decision.

Inspect action outcomes
Use both motion receipts and outcome images.

Reaching a pose does not prove that an object was grasped. Pose errors and stopping reasons distinguish small residual errors from rejected or stalled motion; outcome images reveal the effect on the object. Together, they guide motion refinement or a change of strategy.

Keep useful context
Replace old images with text while retaining instructions and complete interactions.

Old images become text placeholders in batches, preserving recent views and a stable prompt prefix for cache reuse. If more space is needed, we remove older complete interactions, keeping calls paired with results and protecting essential instructions and pending calls.

Figure 5 | Six mechanisms, shown step by step. Scenes and motion traces are schematic illustrations of the paper’s design choices.

3. Evaluation

3.1 Simulation Evaluation on RoboDojo

Benchmark and protocol. We use RoboDojo to test open-weight models and compare harnesses across 42 tasks in five dimensions: generalization, precision, long-horizon execution, memory, and open-vocabulary instruction following. Our main configurations use three seeds and two layouts per seed, giving 252 episodes each. The 12 generalization tasks pair standard and randomized layouts. Score credits partial progress; success rate (SR) counts fully successful episodes. We average within each task, then within each dimension, and weight the five dimensions equally. Each episode has a three-hour wall-clock limit: in four-hour runs, aggregate performance largely plateaued by the third hour.

Baselines. With Qwen3.8-Flash-Next as the planner, we compare against GAP (Hybrid)10, which incorporates π0.5, plus RoboProbe and RPent11 in their default configurations. Published Astra–RoboProbe results and VLA / WAM entries from the RoboDojo leaderboard provide broader context.

Results. Qwen with uni-robot-agent reaches 31.39 Score / 25.00% SR, compared with 21.01 / 16.11% for GAP (Hybrid), the strongest Qwen baseline. DeepSeek reaches 27.11 / 21.39%, extending the result to another open-weight model; Astra reaches 51.90 / 46.67%.

Different capability profiles. Qwen and DeepSeek perform better on memory and open-vocabulary tasks than on precision and long-horizon execution. Conversely, Xiaomi-Robotics-112 and OpenWAM-α13 score 38.39 and 34.93 on long-horizon tasks, but 7.81 and 10.41 on memory. These configurations have complementary strengths; precise manipulation and long-horizon planning remain challenges for current foundation models.

Table 1 | Results on RoboDojo. Each cell shows SR above Score (0–100); Avg. weights the five capabilities equally. Our configurations use 252 episodes each; published baselines retain their original protocols. †GAP (Hybrid) incorporates π0.5.Our primary results+ depth · reference
ModelHarnessGen.Prec.Long-hor.Mem.OpenAvg.↑
VLA & WAM
π0.5–8.17/13.385.50/12.4014.67/23.544.67/5.891.67/1.986.93/11.44
Hy-Embodied-0.5-VLA14–8.39/11.788.00/13.8114.92/25.7412.11/13.370.58/0.658.80/13.07
OpenWAM-α–14.83/20.719.25/18.4525.33/34.939.11/10.411.08/1.4111.92/17.18
Xiaomi-Robotics-1–17.00/23.5418.83/26.6923.67/38.396.56/7.813.58/3.9413.93/20.07
Agent
GPT-6 AstraRoboProbe30.50/33.364.00/12.658.25/21.4538.67/43.0431.00/34.3622.48/28.97
GPT-6 Astrauni-robot-agent31.94/37.1520.83/28.7539.58/50.3180.56/80.5660.42/62.7146.67/51.90
GPT-6 Astrauni-robot-agent + depth43.06/47.8529.17/36.0452.08/60.3175.00/75.4262.50/64.1752.36/56.76
Qwen3.8-Flash-NextRoboProbe0.00/0.350.00/0.830.00/3.750.00/0.004.17/4.580.83/1.90
Qwen3.8-Flash-NextRPent5.56/9.244.17/8.024.17/12.405.56/5.562.08/3.544.31/7.75
Qwen3.8-Flash-NextGAP (Hybrid)†13.89/19.318.33/16.4618.75/28.8516.67/16.6722.92/23.7516.11/21.01
DeepSeek-V4.1-Flashuni-robot-agent16.67/20.356.25/16.254.17/14.7936.11/38.3343.75/45.8321.39/27.11
DeepSeek-V4.1-Flashuni-robot-agent + depth18.06/21.8812.50/19.7910.42/25.3141.67/45.5637.50/38.5424.03/30.22
Qwen3.8-Flash-Nextuni-robot-agent18.06/21.602.08/10.216.25/23.7552.78/54.7245.83/46.6725.00/31.39
Qwen3.8-Flash-Nextuni-robot-agent + depth22.22/25.2818.75/26.3514.58/29.6952.78/56.2539.58/41.8829.58/35.89
How stable are the scores?Rollout-count sensitivity

We examine sampling variability in a separate depth-enabled Qwen3.8-Flash-Next evaluation: 50 rollouts for each of 54 task variants, or 2,700 episodes. The variants include standard and randomized versions of the 12 generalization tasks, plus the other 30 tasks. The full-pool Score is 34.59.

Score variability narrows as rollouts per task variant increase from 1 to 50. At six rollouts, the mean is 34.59 and the central 95% subsampling range is 30.65 to 38.51; the full-pool reference is 34.59.
Figure 6 | Score variability with fewer rollouts. The shaded band is the central 95% range across 10,000 random subsets at each sample size; the dashed line marks the full-pool reference. The marker highlights six rollouts per task variant.

With six rollouts per variant, the mean sampled Score is 34.59, with a central 95% range of 30.65–38.51. Taking the first six recorded rollouts, ordered by pass and layout identifiers, instead gives 36.55—an upward difference of 1.96 points. A small sample can recover the aggregate Score on average while individual subsets still vary by several points.

How to read this check. At each sample size, episodes are sampled independently without replacement within each variant, with the five capability dimensions weighted equally. Six rollouts per variant here gives 324 episodes. This is a separate depth-enabled pool, not the main no-depth, three-seed/two-layout evaluation. The band describes variation within the observed pool, not a confidence interval for the main Score of 31.39 or a bias correction to it; it collapses at 50 by construction. Read the sampling values.

3.2 Real-World Evaluation on Real-World Benchmark

Benchmark and protocol. Real Eval tests tabletop manipulation on a physical dual-arm robot through 19 evaluation entries. Five dimensions—spatial understanding, object generalization, instruction following, compositional generalization, and unseen tasks—contribute 20 points each. Task-specific criteria award credit for completed objectives and, where specified, intermediate progress such as grasping before placement.

Baselines. We evaluate uni-robot-agent zero-shot with Astra and Qwen, alongside π0.5 fine-tuned on demonstrations from the same platform and Dream-1 evaluated zero-shot through its public API. Our agents additionally use depth from Intel RealSense D405 cameras, while π0.5 and Dream-1 use RGB only. The five dimensions describe variation relative to the π0.5 training catalogue; our agents receive no task-specific training on these entries.

Results. Astra reaches 80.50 / 100 and Qwen 53.50, compared with 47.75 for fine-tuned π0.5 and 43.50 for Dream-1. These results extend the simulation findings to physical robots with both frontier and open-weight models. Astra scores 20 / 20 on compositional generalization, while unseen tasks remain difficult: both agents score 8 / 20.

Table 2 | Results on Real Eval. Each capability is scored out of 20; total is out of 100. π₀.₅ is fine-tuned on the same platform; Astra and Qwen use uni-robot-agent zero-shot with sensor depth; π₀.₅ and Dream-1 use RGB only.
VLAWAMuni-robot-agent
Capabilityπ₀.₅Dream-1AstraQwen
Spatial understanding11.256.5017.009.00
Object generalization12.5013.5017.5010.00
Instruction following14.009.5018.0013.50
Compositional generalization6.0010.0020.0013.00
Unseen tasks4.004.008.008.00
Total47.7543.5080.5053.50
Per-task scores · 19 evaluation entries
Table 3 | Scores credit completed objectives and intermediate progress; the maximum varies by task.
IDTaskMaxπ₀.₅Dream-1AstraQwen
1Pick and place (fruits)52353
2Fold rag31.751.531
3Bimanual plate pull-out43021
4Insert test tubes into rack42132
5Hang mug42.5142
6Bimanual bottle handover102.55.5104
7Stack bowls (new objects)101087.56
8Stack cups (color order)106185
9Pick and place (assorted objects)6.566.56.56.5
10Stack bowls (given direction)3.5223.52
11Pull out tissue and discard40441
12Stack upward-facing cups44040
13Wipe whiteboard42444
14Arrange cubes (color order)80288
15Close microwave door44444
16Take bowl from microwave40000
17Unfold rag40000
18Stack books40024
19Tower of Hanoi40020
Total10047.7543.5080.5053.50

4. What Capabilities Enable Robot-Use Agents?

The evaluation shows that open models can use robots with the harness. To guide future training, we next ask which capabilities make this possible and where they fall short. We compare generations, families, and model sizes, then connect their failure modes to general-purpose benchmarks.

Performance Across Generations, Families, and Sizes

We evaluate additional models on RoboDojo’s 42 tasks and reuse the main 252-episode results for Qwen3.8-Flash-Next, DeepSeek-V4.1-Flash, and GPT-6 Astra. Most additional models use one seed and one layout per task; Qwen3.8-27B uses three layouts. SR and Score equally weight the five capability dimensions. These are exploratory comparisons, without a claim of statistical significance.

Robot use across thirteen models

SR & Score
Figure 7 | Robot use across thirteen models. Select SR or Score; model order stays fixed. Manufacturer marks identify families, while adjacent labels identify generations and variants. Cohort sizes appear beside each model. GLM uses depth; its Score is specified as zero in the manuscript rather than recovered from native-score records.
Read the data

† GLM Score is set to zero in the manuscript; native-score records are unavailable. Its evaluation includes depth.

ModelEpisodesSR (%)Score
Qwen3.5 Flash420.000.00
Qwen3.5-122B-A10B420.000.00
Qwen3.6 Flash420.000.00
Qwen3.6 Plus420.000.38
Qwen3.7 Flash420.000.00
Qwen3.7 Plus420.000.00
Qwen3.8-27B1268.069.61
Qwen3.8 Flash25225.0031.39
DeepSeek-V4.1-Flash25221.3927.11
GLM-5.3-Flash (+ depth)420.000.00 †
GPT-5.6 Sol4219.1724.04
GPT-6.1 Sol4237.5045.63
GPT-6 Astra25246.6751.90
All chart values and exact recorded curves (JSON)

Robot use predates the latest generation. GPT-5.6 Sol reaches 19.17% SR / 24.04 Score; GPT-6.1 Sol improves to 37.50% / 45.63. The latest generation improves performance substantially, but the ability was already present in earlier GPT models.

The pattern differs across families. Qwen3.5–3.7 Flash, Qwen3.6–3.7 Plus, and Qwen3.5-122B record zero SR. Qwen3.8-27B and Flash-Next reach 8.06% and 25.00%, respectively. Yet a recent generation alone is not sufficient: GLM-5.3 Flash also records zero SR.

Larger is not always more capable. Within Qwen3.8, Flash-Next (125B) outperforms 27B. Across generations, however, 27B completes tasks while earlier 122B and >150B models do not. This motivates the paper’s hypothesis that changes in training distribution matter beyond parameter count.

Four Major Failure Modes

Trajectory inspection reveals four recurring difficulties: interpreting the scene, locating corresponding features across views, computing poses correctly, and choosing physically appropriate interactions. The analysis spans thirteen models, with available measurements shown for each failure type.

Where robot interactions break down

Trajectory analysis
Figure 8 | Where robot interactions break down. Points pair audited error rates with the paper’s model-performance Scores. General vision counts reviewed trials; grounding counts triangulation calls; geometry code counts executed calculations; embodied reasoning counts reviewed pick-and-place tasks with uncorrected, inappropriate grasp proposals assessed by GPT-6 Astra. Dashed curves use the paper’s reciprocal fits, drawn only across the observed error-rate range. They describe model-level associations, not causal effects. Missing or no-call observations are omitted, not plotted as zero. Qwen3.8-27B’s computation rate is based on a visible subset of calculations. Astra’s grounding and computation rates are supplied without counts; its visual-understanding rate is unavailable.
Read the data
ModelFailure typeErrors / reviewedRate (%)
Qwen3.5 FlashVisual understanding5 / 4112.20
Qwen3.5 122B-A10BVisual understanding2 / 405.00
Qwen3.6 FlashVisual understanding10 / 4223.81
Qwen3.6 PlusVisual understanding9 / 4221.43
Qwen3.7 FlashVisual understanding8 / 4219.05
Qwen3.7 PlusVisual understanding9 / 4121.95
Qwen3.8 27BVisual understanding4 / 1253.20
Qwen3.8 Flash 125BVisual understanding0 / 420.00
DeepSeek V4.1 FlashVisual understandingNone / 0N/A
GLM-5.3 FlashVisual understandingNone / 0N/A
GPT-5.6 SolVisual understanding0 / 420.00
GPT-6.1 SolVisual understanding0 / 420.00
GPT-6 AstraVisual understandingN/AN/A
Qwen3.5 FlashSpatial grounding2 / 2100.00
Qwen3.5 122B-A10BSpatial grounding14 / 1687.50
Qwen3.6 FlashSpatial grounding15 / 1883.33
Qwen3.6 PlusSpatial grounding16 / 3151.61
Qwen3.7 FlashSpatial grounding5 / 683.33
Qwen3.7 PlusSpatial grounding17 / 1894.44
Qwen3.8 27BSpatial grounding536 / 83464.27
Qwen3.8 Flash 125BSpatial grounding58 / 11251.79
DeepSeek V4.1 FlashSpatial grounding109 / 17960.89
GLM-5.3 FlashSpatial grounding0 / 0N/A
GPT-5.6 SolSpatial grounding342 / 58958.06
GPT-6.1 SolSpatial grounding359 / 113831.55
GPT-6 AstraSpatial groundingN/A34.10
Qwen3.5 FlashGeometry code9 / 1181.82
Qwen3.5 122B-A10BGeometry code2 / 450.00
Qwen3.6 FlashGeometry code8 / 1080.00
Qwen3.6 PlusGeometry code26 / 5547.27
Qwen3.7 FlashGeometry code21 / 3461.76
Qwen3.7 PlusGeometry code20 / 3951.28
Qwen3.8 27BGeometry code110 / 21850.46
Qwen3.8 Flash 125BGeometry code169 / 20698.17
DeepSeek V4.1 FlashGeometry code30 / 19721.52
GLM-5.3 FlashGeometry code20 / 17211.63
GPT-5.6 SolGeometry code3 / 4270.70
GPT-6.1 SolGeometry code0 / 2590.00
GPT-6 AstraGeometry codeN/A0.00
Qwen3.5 FlashEmbodied reasoning9 / 4121.95
Qwen3.5 122B-A10BEmbodied reasoning3 / 407.50
Qwen3.6 FlashEmbodied reasoning10 / 4223.81
Qwen3.6 PlusEmbodied reasoning4 / 429.52
Qwen3.7 FlashEmbodied reasoning3 / 427.14
Qwen3.7 PlusEmbodied reasoning1 / 412.44
Qwen3.8 27BEmbodied reasoning1 / 1240.81
Qwen3.8 Flash 125BEmbodied reasoning0 / 420.00
DeepSeek V4.1 FlashEmbodied reasoning3 / 427.14
GLM-5.3 FlashEmbodied reasoning11 / 4226.19
GPT-5.6 SolEmbodied reasoning1 / 422.38
GPT-6.1 SolEmbodied reasoning0 / 420.00
GPT-6 AstraEmbodied reasoning0 / 420.00
All chart values and exact recorded curves (JSON)

General vision. Earlier Flash and Plus models mistake objects and action outcomes—for example, reading zero as nine or treating an inverted cup as upright. These errors affect 12.2–23.8% of reviewed trials, compared with 3.2% for Qwen3.8-27B and none of the reviewed trials for Qwen3.8-Flash or GPT-5.6 Sol.

Spatial grounding. Inconsistent image points produce large reprojection errors, so triangulation rejects the result. Lower rejection rates tend to accompany higher task performance. GPT-6.1 Sol has the lowest rate in this comparison, but 31.5% of its triangulation calls still fail these checks—precise grounding remains a substantial challenge.

Geometry code. Earlier Flash and Plus models reverse coordinate transforms or mix quaternion conventions; confirmed errors affect 47.3–81.8% of their executed calculations. The rates fall to 8.2% for Qwen3.8-Flash, 1.5% for DeepSeek, 0.7% for GPT-5.6 Sol, and zero observed errors for GPT-6.1 Sol. Reliable robot-specific computation is another promising target for improvement.

Embodied reasoning. A correctly computed pose can still be a poor way to grasp an object. We focus on pick-and-place tasks and use GPT-6 Astra to assess whether proposed grasp poses are physically reasonable. GLM leaves inappropriate grasp proposals uncorrected in 26.2% of reviewed tasks, repeatedly failing to reorient its wrist. These errors are less frequent in newer Qwen and GPT models. This is one narrow test of embodied reasoning: Astra still struggles with more complex interactions, including bimanual coordination.

Relationship to General Model Capabilities

Four benchmarks probe complementary parts of this loop: ERQA for embodied reasoning, MMMU-Pro for multimodal understanding, ScreenSpot-Pro for visual grounding, and LiveCodeBench v6 for coding.

General capabilities and robot-use performance

Six-model comparison
Figure 9 | General capabilities and robot-use performance. Each logo marks one model. Select a benchmark or inspect a model for exact values. All six models are evaluated across 42 RoboDojo tasks. Qwen3.8-Flash-Next, DeepSeek and Astra use their 252-episode main results; Qwen3.5–3.7 use 42 episodes each. Dashed lines are ordinary least-squares fits. Correlations describe associations, not causal effects.
Read the data
ModelEpisodesBenchmarkBenchmark (%)RoboDojo Score
Qwen3.5 Flash42ERQA53.750.00
Qwen3.6 Flash42ERQA56.750.00
Qwen3.7 Flash42ERQA57.250.00
Qwen3.8 Flash252ERQA66.7531.39
DeepSeek V4.1 Flash 1017252ERQA65.027.11
GPT-6 Astra252ERQA81.051.90
Qwen3.5 Flash42MMMU-Pro76.070.00
Qwen3.6 Flash42MMMU-Pro72.630.00
Qwen3.7 Flash42MMMU-Pro76.880.00
Qwen3.8 Flash252MMMU-Pro79.7131.39
DeepSeek V4.1 Flash 1017252MMMU-Pro80.7627.11
GPT-6 Astra252MMMU-Pro85.0951.90
Qwen3.5 Flash42ScreenSpot-Pro50.730.00
Qwen3.6 Flash42ScreenSpot-Pro36.810.00
Qwen3.7 Flash42ScreenSpot-Pro41.240.00
Qwen3.8 Flash252ScreenSpot-Pro68.4431.39
DeepSeek V4.1 Flash 1017252ScreenSpot-Pro24.7327.11
GPT-6 Astra252ScreenSpot-Pro92.751.90
Qwen3.5 Flash42LiveCodeBench v674.810.00
Qwen3.6 Flash42LiveCodeBench v677.10.00
Qwen3.7 Flash42LiveCodeBench v678.630.00
Qwen3.8 Flash252LiveCodeBench v686.8231.39
DeepSeek V4.1 Flash 1017252LiveCodeBench v687.7927.11
GPT-6 Astra252LiveCodeBench v683.2151.90
All chart values and exact recorded curves (JSON)

All four associations are positive: Pearson r = 0.97 for ERQA, 0.93 for MMMU-Pro, 0.69 for ScreenSpot-Pro, and 0.75 for LiveCodeBench. The pattern is consistent with robot use drawing on several abilities together: interpreting observations, identifying targets, reasoning about consequences, and expressing decisions as code.

The strongest associations are with embodied reasoning and multimodal understanding. ScreenSpot-Pro has a weaker relationship, possibly because its 2D screen grounding differs from the 3D geometry of manipulation. These associations do not isolate the contribution of any one capability. Together with the failure analysis, they suggest that future training should jointly improve visual understanding, spatial grounding, geometric coding, and embodied reasoning—for example, by constructing training tasks from robot experience that target each capability.

5. Analysis of Privileged Information

Model capabilities explain part of the picture; the information and tools available to an agent also matter. We next examine how performance changes without depth or camera calibration, and whether it depends on predefined skills. These comparisons test the support a harness needs to provide under realistic conditions.

RQ1: How Does Depth Affect Spatial Understanding for Robot Use?

Recognizing an object in an image does not tell the agent where to move the robot. We compare three information levels: RGB with depth and calibration, RGB with calibration only, and RGB alone. Depth and calibration allow direct pixel-to-world localization. Calibration alone supports multi-view triangulation, but the agent must find corresponding image points. Without either, it must infer spatial relationships from images and adjust movements using action feedback. Robot proprioception remains available throughout.

We evaluate both models on the same 42 benchmark tasks with skills enabled. We use three seeds and two layouts per task for Qwen (252 episodes), and one seed and one layout per task for Astra (42 episodes) to limit evaluation costs. The Astra geometry comparison is separate from its 252-episode main result.

With calibration retained, agents can move the camera and triangulate corresponding points across views. Astra’s Score changes from 55.83 to 55.04 without depth, retaining 98.6%; Qwen falls from 35.89 to 31.39, retaining 87.5%. This contrast suggests that Astra is more effective at recovering spatial information when direct depth measurements are unavailable.

Progress with less geometric information

Figure 10 | Progress with less geometric information. Recorded progress through 771 tool calls across 42 benchmark tasks, with skills enabled. Qwen uses 252 episodes per condition and Astra 42.
Read the data
Model / conditionEpisodesScore
Astra / depth + calibration4255.83
Astra / calibration only4255.04
Astra / RGB only4231.29
Qwen / depth + calibration25235.89
Qwen / calibration only25231.39
Qwen / RGB only2529.13
All chart values and exact recorded curves (JSON)

How much does calibration still matter?

Figure 11 | How much does calibration still matter? Final Scores across the same 42 benchmark tasks, with skills enabled. Qwen uses 252 episodes per condition and Astra 42.
Read the data
Model / conditionEpisodesScore
Astra / depth + calibration4255.83
Astra / calibration only4255.04
Astra / RGB only4231.29
Qwen / depth + calibration25235.89
Qwen / calibration only25231.39
Qwen / RGB only2529.13
All chart values and exact recorded curves (JSON)

Removing calibration too lowers Astra to 31.29 Score and Qwen to 9.13. With RGB-only input, Astra completes 10 of 42 tasks. Astra’s higher Score and wider task coverage suggest stronger spatial perception from visual observations alone, although both models lose substantial performance.

The paper’s appendix examines two successful Qwen episodes without depth or predefined skills. The agent writes and executes its own geometry code, triangulating tube positions across views in one case and testing candidate table heights against image evidence in another. Both cases connect RGB-based localization to successful manipulation.

RQ2: How Important Is the Skill Library?

A reusable harness should support varied tasks through shared tools, without depending on a handcrafted procedure for each task. We test whether the results mainly reflect benchmark-specific skill guidance by removing Qwen’s access to both the robot skill library and skill writing. Task instructions, environment descriptions, and standard tools remain available.

Without skills, Qwen reaches 30.31 Score / 24.58% SR, compared with 31.39 / 25.00% with skills. It retains 96.6% of the Score, with an SR gap of only 0.42 percentage points. This small aggregate gap suggests that effective task execution is not simply a consequence of the predefined skill library.

Skills help, but are not a prerequisite

Figure 12 | Skills help, but are not a prerequisite. Qwen, 252 episodes per setting, with calibration and without depth. Whiskers show approximate 95% bootstrap confidence intervals.
Read the data
ConditionCapabilityScoreSR (%)
With skillsOverall31.3925.00
With skillsGeneralization21.6018.06
With skillsMemory54.7252.78
With skillsPrecision10.212.08
With skillsLong-horizon23.756.25
With skillsOpen-vocabulary46.6745.83
Without skillsOverall30.3124.58
Without skillsGeneralization23.2620.83
Without skillsMemory47.7841.67
Without skillsPrecision16.888.33
Without skillsLong-horizon26.7716.67
Without skillsOpen-vocabulary36.8835.42
All chart values and exact recorded curves (JSON)

For VLA and WAM policies, demonstrations can communicate requirements that are difficult to specify through text alone. Language-capable agents can also interpret explicit textual rules. Removing predefined skills does not remove the need for a clear task specification—a distinction that matters when designing benchmarks and interpreting generalization.

6. Speed and Efficiency

A harness intended for future training must also make repeated interaction practical. Final scores alone do not tell us how much interaction an agent needs or what it costs. We relate progress to tokens, API price, and elapsed time, then compare serial and pipelined execution.

RQ1: How Does Robot-Use Performance Relate to Token Usage and API Cost?

An agent can spend many interactions locating an object, testing a grasp, or recovering from an unsuccessful action. We analyze the same 252-episode cohorts as the main results to compare the resources spent with the progress achieved.

Astra reaches 51.90 Score with the fewest tokens, but costs $21.29 per episode. Qwen reaches 31.39 at $1.09, outperforming DeepSeek with fewer tokens at a slightly higher API cost. The substantial price gap highlights the value of open models for affordable robot use. Costs use reference API rates from OpenAI15, Alibaba Cloud16, and DeepSeek17.

Score versus tokens and API cost

Figure 13 | Score versus tokens and API cost. Episode means, with 252 episodes per model. Model logos identify each configuration. Ringed logos are Pareto-efficient; unringed logos are dominated on that resource axis. Dashed links connect frontier observations, not predicted performance between them.
Read the data
ModelEpisodesScoreMean tokens (M)Mean cost (USD)
Qwen3.8-Flash-Next25231.3937.6682271.090718
DeepSeek-V4.1-Flash25227.1148.7616350.942606
GPT-6 Astra25251.9016.77886821.292319
All chart values and exact recorded curves (JSON)

Final Score hides when useful progress occurs. The recorded curves first reach Score 25 at approximately 7.4M tokens for Astra, 23.5M for Qwen, and 49.9M for DeepSeek. Qwen reaches that level with less than half DeepSeek’s token usage.

When does interaction turn into progress?

Figure 14 | When does interaction turn into progress? The main plot enlarges the selected interval; drag the pale green window or its edges below to explore the full range. Both views use linear axes. The faint dashed reference marks Score 25. Score is aggregated from recorded intermediate outcomes for the same 252-episode cohorts as the resource comparison, retaining each episode’s latest observed Score.
Read the data
ModelEpisodesFinal ScoreTokens to Score ≥ 25 (M)Time to Score ≥ 25 (min)
Qwen3.8-Flash-Next25231.3923.595.5
DeepSeek-V4.1-Flash25227.1149.999.2
GPT-6 Astra25251.907.442.6
All chart values and exact recorded curves (JSON)

The time axis tells a different story. Astra first reaches Score 25 at 42.6 minutes, Qwen at 95.5, and DeepSeek at 99.2. Qwen’s substantially lower token usage yields little elapsed-time advantage at this progress level. Token efficiency alone does not imply faster robot execution.

RQ2: How Does Pipelining Affect Execution Efficiency?

Pipelining helps when the agent can prepare independent work while an action runs. We compare five selected tasks using Astra with medium reasoning effort, one episode per configuration, and matching seeds and layouts. Episode duration excludes simulator initialization.

The pipelined runs are shorter on all five tasks, with observed reductions of 4.8% to 73.7%. Swapping T blocks takes 12.5 versus 34.7 minutes; pressing buttons by number takes 13.7 versus 25.0 minutes.

Episode duration with and without pipelining

Figure 15 | Episode duration with and without pipelining. Pipelined and serial execution on five selected tasks, with one episode per configuration and task. Durations exclude simulator initialization.
Read the data
TaskPipelined (min)Serial (min)
Swap T blocks12.4834.73
Press by number13.6624.95
Conveyor pickup15.7259.73
Align blocks20.1830.17
Solve equation22.0123.11
All chart values and exact recorded curves (JSON)

API latency, resource sharing, and different action sequences also affect duration. The comparison shows shorter observed runs, but does not attribute the entire reduction to scheduling alone.

7. Conclusion

uni-robot-agent combines shared interfaces, replaceable components, efficient execution, and structured feedback for robot use. Astra achieves strong results in simulation and on physical robots. With the open-weight Qwen3.8-Flash-Next, the harness reaches 31.39 Score on RoboDojo, exceeding the published Astra–RoboProbe result of 28.97.

Precise manipulation and reliable long-horizon execution remain challenges. These results motivate further work on open-weight models that can run locally and learn from robot experience. The harness provides a common foundation for collecting that experience across environments, developing training methods, and evaluating their effects on agent behavior and task performance.

Citation

@article{yuan2026democratizing,
  title={Democratizing Robot-Use Agents: A Unified Harness with Open Models},
  author={Yuan, Jiarui and Zuo, Yuxin and Huang, Xiaoyu and Zhang, Wenbo and Park, Kwanyoung and Yang, Junlin and Zhang, Tianxiang and Chen, Ruoqu and Ouyang, Yutao and He, Bingxiang and others},
  journal={arXiv preprint arXiv:2611.01560},
  year={2026}
}

References

  1. Chen et al. (2026). RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies.
  2. Qwen Team (2026). Qwen3.8-Flash-Next: A New Architecture, Towards Ultimate Cost-Efficiency.
  3. OpenAI (2026). GPT-6 Astra: A New Generation of Intelligence.
  4. Physical Intelligence (2025). π₀.₅: A Vision-Language-Action Model with Open-World Generalization.
  5. Zhang et al. (2026). An Unexpected Robot Policy: Early Evaluations of GPT-6 Astra on RoboDojo and Beyond.
  6. Dream Labs (2026). Zero-Shot to New Robots: Introducing Dream-1.
  7. RoboDojo Team (2026). RoboDojo Simulation Leaderboard.
  8. Xi et al. (ACL 2025). AgentGym: Evaluating and Training Large Language Model-based Agents across Diverse Environments.
  9. Luo et al. (2025). Agent Lightning: Train ANY AI Agents with Reinforcement Learning.
  10. Su et al. (2026). GPT 6 Astra as an Embodied Policy.
  11. Zhang et al. (2026). Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents.
  12. Xiaomi Robotics Team (2026). Xiaomi-Robotics-1: Scaling Vision-Language-Action Models with over 100K Hours of Real-World Trajectories.
  13. Wang et al. (2026). OpenWAM: An Open, Modular Exploration Towards Systematic World-Action Model Pretraining.
  14. Zhang et al. (2026). Hy-Embodied-0.5-VLA: From Vision-Language-Action Models to a Real-World Robot Learning Stack.
  15. OpenAI. GPT-6 Astra API pricing.
  16. Alibaba Cloud. Qwen3.8-Flash model information and pricing.
  17. DeepSeek. Models and pricing.
Evaluation

Evaluating robot-use agents

Evaluation protocols for resource budgets, episode termination, and task-specific knowledge.

Explore the evaluation