RDJ a suite of the exam · Two arms
RoboDojo
Bimanual manipulation over five capabilities: generalization, memory, precision, long horizons, open tasks.
The simulation part of RoboDojo's benchmark for generalist manipulation policies: two ARX X5 arms with parallel grippers at a table. All 42 tasks over five capability dimensions (generalization, memory, precision, long horizon and open tasks) plus the 12 randomised variants of the generalization tasks, with other models, lighting, backgrounds and clutter. Each task is one frozen evaluation layout, in a privileged and a standard mode. A privileged answer is replayed up to three times, each in a fresh process, and counts as solved if any replay succeeds: the physics of six tasks (cloth, liquids, toast) does not replay bit for bit.
Three prompt versions. RoboDojo's 37 core tasks in the Standard Setting, run by GPT-6 Astra and Claude Fable 5.2 with three prompts; scenes, step limits and success checkers are the same.
- GPT-6 AstraClaude Fable 5.2
- RoboDojo VLA prompt: RoboDojo's one-sentence task description.16% (6/37)57% (21/37)
- RoboDojo agent prompt: the same sentence plus the task recipe and the return-to-start note from RoboDojo's own agent evaluation.46% (17/37)65% (24/37)
- Our prompt: the task stated together with every condition RoboDojo's success checker tests, including that both arms end near their start pose.57% (21/37)76% (28/37)
The leaderboard uses our prompt.
Example: Fill Egg Holder
Task: Place the four eggs from the basket into the egg holder, then close the lid.
Task: Place the four eggs from the basket into the egg holder, then close the lid.
Description. There is a woven basket containing four eggs and an egg holder. The robot needs to pick up the eggs one by one, place all four into the egg holder, and then close the lid.
Scoring.
- 0: No egg is placed into the egg holder.
- 10: Any one egg is inside the egg holder and close to the holder bottom, with the gripper open.
- 25: Any two eggs are placed correctly into the egg holder.
- 40: Any three eggs are placed correctly into the egg holder.
- 90: All four eggs are placed in the holder, but the lid is not fully closed yet.
- 100: All four eggs are in the holder, the holder lid is correctly and fully closed, and the robot returns to origin.
Most of these tasks also score the arms themselves: the work does not score until both arms are back within 0.15 m and 20 degrees of the pose they began the episode in, with the objects left as the task asked. Getting home is the last step of the task rather than tidying up afterwards, so keep enough env steps in hand to make it.
Task: Move the four eggs from the egg basket into the egg holder (an articulated tray with a lid), close the lid, and return the arm home.
Success. All of these must hold at the same action boundary, for each of the four eggs (any egg in any cup):
- In the holder: the egg's centre … seen from above, inside the holder's outline … and higher than that outline's lowest point.
- Seated in a cup: the egg's centre strictly between 0.01 and 0.04 m higher than the carton's floor point … An egg resting in a cup passes; an egg on top of the closed lid, on the rim or held above the carton fails.
- Lid closed: the lid's hinge more than 90 % of the way along its range toward closed …
- Both arms back: each arm's end-effector within 0.15 m of its start position on each of x, y and z separately, and within 20 deg of its start orientation. The grippers need not be open.
Full prompts also include the robot client, rules and time limit, the same in all three.
| Origin | adapted from the upstream benchmark |
|---|---|
| Simulator | Isaac Lab 2.3 / Isaac Sim 5.1 |
| Robot | Two ARX X5 arms |
| Tasks | 54 tasks · 108 tests |
| Agent time | 60 min |
On a real robot
Tasks
54 tasks · 108 tests