RLB  a suite of the exam · One arm

RoboLab

Tabletop tasks that test language grounding, from colours and counts to relations and order.

53 of its 120 tabletop manipulation tasks for a fixed-base arm, built to test language grounding and goal specification rather than locomotion: colour and size references, counting, spatial relations, sorting, ordered stacking, reorientation. Success is a composable predicate over object state, with partial credit from scored subtasks.

Originadapted from the upstream benchmark (v0.3.1 (2026-08-11))
SimulatorIsaac Lab 2.3.2 / Isaac Sim 5.1 (NVIDIA)
RobotFranka + Robotiq 2F-85
Tasks53 tasks · 106 tests
Agent time60 min
01

Results

6 tasks
1 Claude Fable 5.2Claude Code · high 33%2/6
1 GPT-6.1 SolCodex · high 33%2/6
1 GPT-6.2 AstraCodex · high 33%2/6
4 Claude Opus 5.5Claude Code · high 17%1/6
4 Gemini 3.8 FlashAntigravity · high 17%1/6
6 DeepSeek V4.1 FlashDeepSeek Harness · high 0%0/6
6 Gemini 4 ArgonAntigravity · high 0%0/6
6 GLM 5.3 FlashClaude Code · high 0%0/6
6 GPT-6 AstraCodex · high 0%0/6
6 Kimi 3.1 ProClaude Code · high 0%0/6
02

Tasks

53 tasks · 106 tests