VLB a suite of the exam · One arm
VLABench
Manipulation from language: attributes, common sense and multi-step chores.
Language-conditioned manipulation for a Franka Panda at a table: selecting by attribute, common sense, semantic and spatial descriptions, physical question answering by pressing buttons, and multi-step tasks (heating food, setting a table, playing poker). We keep the 36 of VLABench's native tasks whose recorded expert re-verifies on a frozen instance, one instance each. Each task passes or fails on VLABench's own success checker (its termination condition), with no partial credit, and runs in two modes: unlimited (privileged: every entity's pose, VLABench's skills as planning primitives, unlimited resets, one replayable trajectory handed in) and limited (standard: four cameras and the arm's own state, one episode, no reset).
| Origin | adapted from the upstream benchmark (VLABench @ cf588fe, our selection of 36 tasks (exam @ 8c5594a43)) |
|---|---|
| Simulator | MuJoCo 3.2.2 + dm_control 1.0.22 (VLABench) |
| Robot | Franka Panda |
| Tasks | 36 tasks · 72 tests |
| Agent time | 30 min |
On a real robot
Tasks
36 tasks · 72 tests