ESW a suite of the exam · One arm, humanoid and two arms
EmbodiedSWE
Long-horizon assembly, packing, cutting and loco-manipulation, solved by a program the agent writes.
Twenty-four tasks of EmbodiedSWE's robot benchmark on Isaac Lab: install a graphics card and RAM, drive the motherboard's bolts, screw in a light bulb, thread a nut, assemble a side table or a robot arm's elbow with two arms, pack tools and pencils, dice and slice food with a knife, make a capsule coffee, flip bread with a spatula, dose with a syringe, and stack, sort and carry objects with a Unitree G1, standing or walking. The agent delivers a program that drives the robot through the task in a fresh simulator built from the frozen scene.
| Origin | adapted from the upstream benchmark |
|---|---|
| Simulator | Isaac Lab 2.3.2 on Isaac Sim 5.1 (GPU PhysX) |
| Robot | Franka Panda, Two Franka Panda arms, Unitree G1 with Dex3 hands |
| Tasks | 24 tasks · 48 tests |
| Agent time | 240 min |
On a real robot
12 tasks
1
Claude Fable 5.2Claude Code · high
67%8/12
1
GPT-6.2 AstraCodex · high
67%8/12
3
Claude Opus 5.5Claude Code · high
42%5/12
3
GPT-6 AstraCodex · high
42%5/12
3
GPT-6.1 SolCodex · high
42%5/12
6
Gemini 4 ArgonAntigravity · high
33%4/12
7
Gemini 3.8 FlashAntigravity · high
17%2/12
8
DeepSeek V4.1 FlashDeepSeek Harness · high
0%0/12
8
GLM 5.3 FlashClaude Code · high
0%0/12
8
Kimi 3.1 ProClaude Code · high
0%0/12
02
Tasks
24 tasks · 48 tests