VLB  a suite of the exam · One arm

VLABench

Manipulation from language: attributes, common sense and multi-step chores.

Language-conditioned manipulation for a Franka Panda at a table: selecting by attribute, common sense, semantic and spatial descriptions, physical question answering by pressing buttons, and multi-step tasks (heating food, setting a table, playing poker). We keep the 36 of VLABench's native tasks whose recorded expert re-verifies on a frozen instance, one instance each. Each task passes or fails on VLABench's own success checker (its termination condition), with no partial credit, and runs in two modes: unlimited (privileged: every entity's pose, VLABench's skills as planning primitives, unlimited resets, one replayable trajectory handed in) and limited (standard: four cameras and the arm's own state, one episode, no reset).

Originadapted from the upstream benchmark (VLABench @ cf588fe, our selection of 36 tasks (exam @ 8c5594a43))
SimulatorMuJoCo 3.2.2 + dm_control 1.0.22 (VLABench)
RobotFranka Panda
Tasks36 tasks · 72 tests
Agent time30 min
01

Results

2 tasks
1 Claude Fable 5.2Claude Code · high 50%1/2
1 Claude Opus 5.5Claude Code · high 50%1/2
1 GPT-6.1 SolCodex · high 50%1/2
1 GPT-6.2 AstraCodex · high 50%1/2
5 DeepSeek V4.1 FlashDeepSeek Harness · high 0%0/2
5 Gemini 3.8 FlashAntigravity · high 0%0/2
5 Gemini 4 ArgonAntigravity · high 0%0/2
5 GLM 5.3 FlashClaude Code · high 0%0/2
5 GPT-6 AstraCodex · high 0%0/2
5 Kimi 3.1 ProClaude Code · high 0%0/2
02

Tasks

36 tasks · 72 tests