MOL a suite of the exam · One arm and mobile robot
MolmoSpaces
Zero-shot episodes in unseen houses, for a Franka FR3 and an RB-Y1 mobile robot.
Ai2's zero-shot evaluation episodes for robot policies in unseen houses: a Franka FR3 picks, places (into a receptacle, next to an object, by colour), opens and closes fixtures, and a Rainbow RB-Y1 mobile manipulator navigates to an object and opens a room door. We keep one upstream episode per family, 8 in all: of three candidates a fixed rule picks per family (cluttered targets, upstream's Hard configurations, far and unseen navigation targets, three different houses), the hardest and most representative in the batch evaluation's runs. Each task passes or fails on upstream's own judge_success() at the end of the episode, with no partial credit, and runs in two modes: unlimited (privileged: the simulator's full state, upstream's kinematics, unlimited resets, one replayable trajectory handed in) and limited (standard: the robot's own cameras with depth and its proprioception, one episode, no reset).
| Origin | adapted from the upstream benchmark (molmo-spaces 0.2.9, benchmark molmospaces-bench-v2/20260415; our selection of 8 episodes (exam @ e46be110c,) |
|---|---|
| Simulator | MuJoCo 3.5.0 (molmo-spaces 0.2.9) |
| Robot | Franka + Robotiq 2F-85, Rainbow RB-Y1 |
| Tasks | 8 tasks · 16 tests |
| Agent time | 60 min |
On a real robot
Tasks
8 tasks · 16 tests