ESW  a suite of the exam · One arm, humanoid and two arms

EmbodiedSWE

Long-horizon assembly, packing, cutting and loco-manipulation, solved by a program the agent writes.

Twenty-four tasks of EmbodiedSWE's robot benchmark on Isaac Lab: install a graphics card and RAM, drive the motherboard's bolts, screw in a light bulb, thread a nut, assemble a side table or a robot arm's elbow with two arms, pack tools and pencils, dice and slice food with a knife, make a capsule coffee, flip bread with a spatula, dose with a syringe, and stack, sort and carry objects with a Unitree G1, standing or walking. The agent delivers a program that drives the robot through the task in a fresh simulator built from the frozen scene.

Originadapted from the upstream benchmark
SimulatorIsaac Lab 2.3.2 on Isaac Sim 5.1 (GPU PhysX)
RobotFranka Panda, Two Franka Panda arms, Unitree G1 with Dex3 hands
Tasks24 tasks · 48 tests
Agent time240 min
01

Results

12 tasks
1 Claude Fable 5.2Claude Code · high 67%8/12
1 GPT-6.2 AstraCodex · high 67%8/12
3 Claude Opus 5.5Claude Code · high 42%5/12
3 GPT-6 AstraCodex · high 42%5/12
3 GPT-6.1 SolCodex · high 42%5/12
6 Gemini 4 ArgonAntigravity · high 33%4/12
7 Gemini 3.8 FlashAntigravity · high 17%2/12
8 DeepSeek V4.1 FlashDeepSeek Harness · high 0%0/12
8 GLM 5.3 FlashClaude Code · high 0%0/12
8 Kimi 3.1 ProClaude Code · high 0%0/12
02

Tasks

24 tasks · 48 tests