ABC  a suite of the exam · Two arms

ABC Sim

Two YAM arms at a table: sort, count, spell, pour, sweep, hang a mug and balance a ball on a tray.

Twenty-nine tasks of ABC Sim, the MuJoCo simulator of ABC (Scalable Behavior Cloning with Open Data, Training, and Evaluation), every task its catalogue scores: two I2RT YAM arms side by side at a table, driven by joint-position targets. Throw or put bottles in a bin, pick parts off a moving conveyor, put exactly N objects in an opaque box, grab one item from clutter, search drawers, pour, hang a mug, turn mugs upright, sweep scraps, load a dish rack, sort nuts and bolts or lego blocks, set up chess pieces, spell words with letter blocks, hand an item from one gripper to the other and keep a ball balanced on a tray held with both arms. Each task is one frozen scene from an upstream demonstration; the goal must hold for a second to count.

Originadapted from the upstream benchmark
SimulatorABC Sim on MuJoCo 3.6.0
RobotTwo I2RT YAM arms
Tasks29 tasks · 60 tests
Agent time60 min
01

Results

Different models were evaluated on different tasks, so the results are not fully comparable.

29 tasks
· Claude Opus 5.5Claude Code · high 100%1/1
· GPT-6 AstraCodex · high 100%1/1
· GPT-6 LunaCodex · xhigh 21%6/29
· DeepSeek V4.1 FlashDeepSeek Harness · high 0%0/1
· Gemini 3.8 FlashAntigravity · high 0%0/1
· Gemini 4 ArgonAntigravity · high 0%0/1
· GLM 5.3 FlashClaude Code · high 0%0/1
02

Tasks

29 tasks · 60 tests