MOL  a suite of the exam · One arm and mobile robot

MolmoSpaces

Zero-shot episodes in unseen houses, for a Franka FR3 and an RB-Y1 mobile robot.

Ai2's zero-shot evaluation episodes for robot policies in unseen houses: a Franka FR3 picks, places (into a receptacle, next to an object, by colour), opens and closes fixtures, and a Rainbow RB-Y1 mobile manipulator navigates to an object and opens a room door. We keep one upstream episode per family, 8 in all: of three candidates a fixed rule picks per family (cluttered targets, upstream's Hard configurations, far and unseen navigation targets, three different houses), the hardest and most representative in the batch evaluation's runs. Each task passes or fails on upstream's own judge_success() at the end of the episode, with no partial credit, and runs in two modes: unlimited (privileged: the simulator's full state, upstream's kinematics, unlimited resets, one replayable trajectory handed in) and limited (standard: the robot's own cameras with depth and its proprioception, one episode, no reset).

Originadapted from the upstream benchmark (molmo-spaces 0.2.9, benchmark molmospaces-bench-v2/20260415; our selection of 8 episodes (exam @ e46be110c,)
SimulatorMuJoCo 3.5.0 (molmo-spaces 0.2.9)
RobotFranka + Robotiq 2F-85, Rainbow RB-Y1
Tasks8 tasks · 16 tests
Agent time60 min
01

Results

2 tasks
1 Claude Fable 5.2Claude Code · high 100%2/2
1 Claude Opus 5.5Claude Code · high 100%2/2
1 Gemini 4 ArgonAntigravity · high 100%2/2
1 GPT-6 AstraCodex · high 100%2/2
1 GPT-6.1 SolCodex · high 100%2/2
1 GPT-6.2 AstraCodex · high 100%2/2
7 DeepSeek V4.1 FlashDeepSeek Harness · high 50%1/2
7 GLM 5.3 FlashClaude Code · high 50%1/2
7 Kimi 3.1 ProClaude Code · high 50%1/2
10 Gemini 3.8 FlashAntigravity · high 0%0/2
02

Tasks

8 tasks · 16 tests