One line of code. 1,156 tasks. 35 suites, 47 robots.
What robotic tasks can coding agents solve?
One command connects a coding agent to 1,156 tasks in 35 suites, on robot arms, dexterous hands, humanoids, quadrupeds and musculoskeletal models. The agent chooses its own control approach: direct control, code to control, or train a policy to control.
Under what physical-world constraints?
The Privileged Setting gives the full simulator state and unlimited resets. The Standard Setting gives only the robot’s sensors and, by default, no resets, and its Realtime variant keeps the world running while the agent reasons.
Where do they fail?
We label the stage at which each failed attempt goes wrong and compare failures across agents, settings and stages. Every run is published with its full trace and a replay. The full analysis is in Where Do Today’s Agentic Robots Succeed and Fail?
RoboPaint · Oil, The Starry Nightreference solution
RoboARC · Untie a knotreference solution
RoboDynamicExam · Launch, catch, reloadreference solution
Embodied Olympics - Fun · Catch and shootGPT-6 Astra · Privileged
Embodied Olympics - Match · Fencingagent solution · wins 15:13
DexHandExam · Assemble a hamburgerreference solution
RoboEngineer · Weave between polesGPT-6 Astra · Standard
Embodied Safety Exam · Set the table on a place matreference solution
iSpatialGym · Match a goal imageGPT-6 Astra · Privileged
01Motivation
Coding agents have passed many exams in the digital world: they resolve issues in real repositories1, complete hard tasks in the terminal2 and write efficient GPU kernels3. But what is the first exam they must pass to enter the physical world? Robotics adds perception, contact and irreversible actions in a world that may keep moving while the agent thinks. Robots acquire new skills either by manual programming or by learning from data, as in vision-language-action models4–6. Coding agents suggest a third way (Figure 2), often called agentic robotics: they turn instructions into executable code7 and revise their plans through automated experimentation. We ask what robotic tasks general-purpose coding agents can solve, under what physical-world constraints, and where they fail, on tasks far beyond pick-and-place (Figure 3). The Embodied First Exam answers this with tasks and up to three evaluation settings per task, which separate what an agent can do with full simulator access from what it can do through the robot’s own sensors, without resets and on a running clock.
Learn from data
A policy learns from many demonstrations. Good at the skills it was trained on, but a new task needs new data.
Program it by hand
An engineer writes controllers and planners, tuned for one robot and one job.
A coding agent is the policy
Direct control, code to control or train a policy to control, from a plain-language instruction.
Untie a knot, take a ring off a horseshoe, free a hedgehog from its cage, make a scene match a picture of the goal.
RoboARC · iSpatialGym · RoboWits · KinDERFence, play table tennis or soccer against another robot; catch a chest pass and shoot it through the hoop.
Embodied Olympics - Match · Embodied Olympics - FunThe clock keeps running while the agent thinks: catch a thrown object, return a ball, react to an opponent.
RoboDynamicExam · Embodied Olympics - Match · Embodied Olympics - Fun · HumanoidBench · BiGym 2.0Sprint, jump, dunk and backflip with a humanoid; walk and balance on four legs; control muscles instead of motors.
Embodied Olympics - Fun · RoboDynamicExam · HumanoidBench · MuJoCo Playground · MyoSuiteTune a controller in a public simulator, deploy it to a private one whose dynamics differ, and revise it from the returned measurements.
RoboEngineerDexterous hands and tools, long household activities in a full-size house, and safety: carry out harmless requests and refuse ones that would hurt someone.
DexHandExam · DexToolBench · BEHAVIOR-1K · RoboCasa365 · Embodied Safety Exam · ManiGuard02Introduction
The exam is a robotics benchmark for coding agents: it tests them on 1,156 robot tasks under rules fixed in advance. Each task gives a goal in plain language and a frozen scene, and it can be taken in up to three settings, from full access to the simulator to the constraints of a real robot (Figure 4). The agent works in a sandboxed container whose only interface is a command-line shell: it reads files, writes and runs code, and reaches the robot through the interface of the setting. A verifier in its own sandbox grades what physically happened. Every task is packaged the same way, so one command runs different coding agents, each in its own harness, such as Codex or Claude Code. Most robotics benchmarks evaluate learned policies, like vision-language-action models. This exam evaluates the coding agent itself, on 47 robots of eight types.
Tasks and tests
The exam has 1,156 tasks and 2,288 tests in 35 suites, using 47 robots. A task is one frozen scene with one goal, and each pair of a task and a setting is a test. Nine of the suites are new (RoboPaint, RoboARC, RoboDynamicExam, Embodied Olympics - Fun, Embodied Olympics - Match, DexHandExam, RoboEngineer, Embodied Safety Exam and iSpatialGym). The other adapt existing benchmarks, and their scenes, robots and success criteria are what the exam is built on; we are grateful to their authors: BEHAVIOR-1K8, RoboLab9, RoboCasa10, RoboCasa36511, RoboCasa-GR112, MetaWorld+13, VLABench14, HumanoidBench15, MolmoSpaces16, KinDER17, RoboTwin 2.018, MuJoCo Playground19, DexToolBench20, DexJoCo21, MyoSuite22, RoboDojo23, BiGym 2.024, YAMLab25, LIBERO-MAX26, AM-Bench27, RoboWits28, ManiGuard29, AutoBio30, LabUtopia31, EmbodiedSWE and ABC Sim. You can browse the suites or the tasks.
A task in this exam is far larger than a question in a language benchmark. It asks for a complete behavior, such as assembling a piece of furniture or playing a whole match, that combines perception, planning, control and recovery from mistakes, and each attempt gets an engineer's budget: from to of agent time, depending on the suite, and a container with 2 to 16 CPU cores and 4 to 39 GiB of memory, with a GPU where the simulation backend or the task needs one. Tasks within a suite often test similar skills, and current agents already solve many of them.
We therefore report results on a core set of hard tasks ( tests). Experts among the authors and a coding agent selected the core set through trial runs, choosing tasks that are challenging and representative of their suite, and it keeps every task that the authors of a suite asked to keep. The other tasks stay in the exam and can be run with harbor run -d embodied-first-exam@all.
Three settings
A task can be run in up to three settings, and each one is a separate test, scored on its own. The rest of this post uses these three names exactly as defined here.
| Privilegedinside the simulator | Standardreal-robot constraints | Realtimeon a running clock | |
|---|---|---|---|
| Reaches | the simulator directly, in the agent’s container | the robot only, through its API | the robot only, through its API |
| Sees | the full state, every object pose | RGB, depth and joints; no object poses | as in Standard |
| Sends | a trajectory.npz | action chunks (T, dim) | action chunks, often from a running program |
| Resets | unlimited | 0 by default; the task’s author can allow more | 0 by default; the task’s author can allow more |
| Clock | plans offline | waits for each command | keeps running while the agent thinks |
| Graded | by a fresh replay | on the live episode | on the live episode |
How one test runs
Figure 5 shows how one test runs, and the list below walks through it.
Completely separate the two claw pieces and briefly hold both still.
- rebuilt bit for bit
- start state checked
- same scene in every setting
- reach
- APIrobot API only
- sees
- RGBdepthjointsno object poses
- sends
- action chunks
(T, dim)
- reach
- inside the simulator
- sees
- full stateresets ∞everything, at will
- sends
- trajectory.npzreplayed fresh
- Question. A test starts from a goal written in words, as you would explain the job to a person, with every convention the agent cannot guess (frames, units, the action layout, the time limit), and from a frozen scene. The scene is rebuilt bit for bit, its start state is checked before the agent begins, and all tests of a task start from the same scene.
- Model and harness. A model runs in its own provider’s harness, such as Codex or Claude Code, or in another provider’s harness through an adapter or a compatible API. The harness runs unchanged and receives the goal as its prompt. It works in a sandboxed container through a command-line shell and can reach only its model’s API and the package registries, with web search off; the Embodied Olympics - Fun tasks are the exception, with open network access.
- The coding agent is the policy. Whatever the agent writes turns what the robot senses into actions. It can control the robot directly, sending commands and choosing the next one after it sees the result; write code that controls the robot, such as a planner or a feedback controller; or train a policy, for example with reinforcement learning, that controls it. The exam leaves the choice to the agent.
- Interface. How the agent’s code reaches the robot depends on the setting. In the Privileged Setting the agent works inside the simulator; in the Standard Setting it reaches the robot only through the robot’s API, and the Realtime Setting is the same with a clock that never stops. In the Standard and Realtime settings the agent cannot reset the episode by default, although a task’s author can allow a fixed number of resets.
- Simulators. Each suite keeps its original simulation setup behind the same interface. The exam covers various simulators, including MuJoCo, Isaac Sim, OmniGibson (built on Isaac Sim), SAPIEN and Genesis, with different robot models running across them.
- Episode trajectory. Every episode is recorded step by step as a rollout: the full simulator state s, the robot’s observation o (camera images and joint readings) and the action a. What the agent itself can read depends on the setting: in the Privileged Setting it reads the full state s, while in the Standard and Realtime settings it sees only the observation o, as with a real robot. The verifier always works from the recorded state.
- Verifier. The verifier runs in its own sandbox with no network and sees only what was recorded. In the Standard and Realtime settings it grades the live episode; in the Privileged Setting it replays the trajectory. The test scores 1 or 0, and the run’s steps, cost and video are logged.
On RoboDojo, how fully the goal states the success check changes the results a great deal. We gave GPT-6 Astra the RoboDojo tasks in the Standard Setting with three versions of the goal text and kept everything else the same (Table 1). RoboDojo’s own one-sentence instructions leave out conditions that its success check tests, above all that both arms end near their start pose, which 47 of the 54 tasks require, so the agent often finishes the visible job and leaves an arm out of place. The text that RoboDojo gave GPT-6 Astra in its own evaluation32 adds a task recipe and a note about the start pose. With that text the agent solves almost as many tasks as with the exam’s goal, which states the success check in full, and most of the remaining differences come from how the manipulation turned out rather than from the wording. The exam’s goals therefore state the conditions they are graded on.
| Goal text | Tasks solved |
|---|---|
| RoboDojo’s one-sentence instruction | 12 / 54 |
| The same sentence, plus the task recipe (task description and scoring stages) and the note that both arms must return to their start pose: the text RoboDojo gave GPT-6 Astra in its own evaluation32 | 33 / 54 |
| The exam’s goal, which states RoboDojo’s success check in full | 37 / 54 |
Three ways to control the robot
An agent can steer the robot itself (direct control), write the code that controls it (code to control), or train a policy that controls it (train a policy to control). The exam does not prescribe which, and agents mix them.
Direct controlAgent⇄Robot
The agent is the controller: it sends a command, reads what the robot sensed, and chooses the next command, step by step, inside its own reasoning loop. Most attempts in the Standard Setting work this way. See it in a real run →
Code to controlCode→Robot
The agent writes a program that turns goals into actions through tools such as inverse kinematics, a motion planner or a feedback controller, and the program drives the robot on its own; between runs the agent reads the outcome and fixes the code. Most attempts in the Privileged and Realtime settings work this way. See it in a real run →
Train a policy to controlPolicy→Robot
The agent trains a policy, for example with reinforcement learning in simulation, and lets the trained policy drive the robot. The policy can be any network, not only a vision-language-action model. This is the rarest of the three. See it in a real run →
Keeping the grading honest
Reference solutions never enter the agent’s container, and we remove the original benchmarks’ solutions from the images. Doing nothing must score 0 on every task, and the reference solution must score 1 wherever one exists. Tasks without one, like most of BEHAVIOR-1K, rely on the original benchmark’s success checker. Before we add a suite, we tell an agent to try to pass without doing the task; if it succeeds anywhere, we hold the suite back until its verifier is fixed.
Two observations from this testing affect how the results should be read. First, models remember deleted solutions: Claude Opus 5.5 rewrote a deleted RoboTwin reference solution from memory, with 26 of its 28 lines identical to the original, and both Opus and GPT-6 Astra rebuilt a deleted HumanoidBench reward function with its original constants, so results on tasks built from public benchmarks may partly reflect memorization. Second, one harness offers tools that run on the provider's servers, outside the container and its network rules, and agents found and used them; these tools are turned off in the exam.
03Overall results
We evaluate coding agents on the core set. An agent is a model in a harness at high reasoning effort, and the harness is part of what we measure: Claude Fable 5.2 and Claude Opus 5.5 in Claude Code, GPT-6 Astra and GPT-6.1 Sol in Codex, Gemini 4 Argon in Antigravity CLI, Kimi 3.1 Pro and Qwen3.8 Max in Claude Code, and three Flash-class models, Gemini 3.8 Flash in Antigravity CLI, GLM 5.3 Flash in Claude Code and DeepSeek V4.1 Flash in DeepSeek Harness. Each agent makes one attempt per test under the same limits, and a test is solved when the verifier passes it. We report each setting on its own, Standard first, as on the leaderboard.
Figure 6. Success rate of each agent on the core set in each setting. Claude Fable 5.2 leads in every setting, and every agent solves by far its largest share of tests in the Privileged Setting. Each bar is the share of a setting’s core tests that one agent solved (the number on top is the percentage).
The numbers
| Agent | Standard | Privileged | Realtime |
|---|---|---|---|
| Claude Fable 5.2 | 50% · 226/456 | 80% · 312/391 | 41% · 22/54 |
| GPT-6 Astra | 34% · 157/456 | 79% · 308/391 | 31% · 17/54 |
| GPT-6.1 Sol | 32% · 147/456 | 73% · 286/391 | 24% · 13/54 |
| Claude Opus 5.5 | 25% · 114/456 | 60% · 233/391 | 7% · 4/54 |
| Gemini 4 Argon | 24% · 110/456 | 43% · 167/391 | 9% · 5/54 |
| Kimi 3.1 Pro | 10% · 46/456 | 38% · 148/391 | 7% · 4/54 |
| Gemini 3.8 Flash | 6% · 29/456 | 24% · 93/391 | 2% · 1/54 |
| Qwen3.8 Max | 4% · 17/456 | 9% · 37/391 | 2% · 1/54 |
| DeepSeek V4.1 Flash | 3% · 14/456 | 17% · 66/391 | 2% · 1/54 |
| GLM 5.3 Flash | 2% · 11/456 | 10% · 38/391 | 0% · 0/54 |
At least one strong agent solves of the Standard tests. Yet in suites, at least half of the Standard tests are solved by none of them, while in the Privileged Setting this holds (Figure 7). Averaged over the strong agents, the success rate is at least 70% in Privileged suites and .
Figure 7. Success rate of each strong agent, by suite (all the core tests of the setting). In the Privileged Setting, one suite has at least half its tests solved by none of the strong agents; in the Standard Setting, seven. Each dot is one agent’s share of a suite’s tests; the gray bar on the right is the share that none of the strong agents solves.
Privileged
Standard
Hardest first. Only suites with at least 5 tests in the setting. Each column lists the suites with at least half their tests solved by none of the strong agents, then the next one below the dashed line. The small label under each name is the kind of task. The IKEA assembly tasks of RoboARC count as their own suite.
The numbers
| Suite | Setting | Tests | Fable 5.2 | GPT-6 Astra | GPT-6.1 Sol | Opus 5.5 | Gemini 4 Argon | None of the strong agents solves |
|---|---|---|---|---|---|---|---|---|
| HumanoidBench | Privileged | 8 | 3 | 3 | 3 | 2 | 2 | 4/8 |
| MyoSuite | Privileged | 13 | 6 | 6 | 7 | 5 | 0 | 6/13 |
| BEHAVIOR-1K | Privileged | 71 | 36 | 36 | 38 | 22 | 6 | 19/71 |
| DexToolBench | Privileged | 20 | 10 | 13 | 10 | 0 | 1 | 4/20 |
| iSpatialGym | Privileged | 8 | 7 | 7 | 7 | 5 | 4 | 1/8 |
| Embodied Olympics - Fun | Privileged | 56 | 49 | 49 | 45 | 45 | 32 | 5/56 |
| RoboDojo | Privileged | 12 | 11 | 10 | 6 | 3 | 3 | 1/12 |
| RoboARC | Privileged | 50 | 46 | 41 | 36 | 38 | 25 | 1/50 |
| DexHandExam | Privileged | 23 | 22 | 22 | 18 | 16 | 8 | 0/23 |
| RoboPaint | Privileged | 22 | 20 | 22 | 20 | 11 | 15 | 0/22 |
| Embodied Olympics - Match | Privileged | 13 | 12 | 11 | 12 | 12 | 12 | 0/13 |
| EmbodiedSWE | Privileged | 12 | 12 | 11 | 11 | 10 | 9 | 0/12 |
| RoboDynamicExam | Privileged | 10 | 10 | 10 | 10 | 10 | 5 | 0/10 |
| BiGym 2.0 | Privileged | 10 | 10 | 10 | 9 | 5 | 7 | 0/10 |
| IKEA assembly | Privileged | 8 | 7 | 6 | 4 | 5 | 2 | 0/8 |
| MuJoCo Playground | Privileged | 7 | 7 | 7 | 7 | 5 | 4 | 0/7 |
| ManiGuard | Privileged | 7 | 7 | 7 | 7 | 7 | 7 | 0/7 |
| RoboLab | Privileged | 6 | 5 | 5 | 4 | 3 | 1 | 0/6 |
| RoboWits | Privileged | 6 | 5 | 5 | 5 | 5 | 5 | 0/6 |
| RoboCasa365 | Privileged | 5 | 5 | 5 | 5 | 4 | 3 | 0/5 |
| HumanoidBench | Standard | 8 | 0 | 1 | 0 | 0 | 0 | 7/8 |
| IKEA assembly | Standard | 8 | 1 | 1 | 0 | 0 | 0 | 7/8 |
| DexToolBench | Standard | 20 | 1 | 2 | 1 | 0 | 0 | 17/20 |
| MyoSuite | Standard | 13 | 5 | 3 | 3 | 3 | 1 | 8/13 |
| BEHAVIOR-1K | Standard | 71 | 26 | 14 | 13 | 5 | 4 | 36/71 |
| RoboPaint | Standard | 22 | 2 | 7 | 6 | 2 | 2 | 11/22 |
| RoboDojo | Standard | 12 | 6 | 3 | 0 | 2 | 4 | 6/12 |
| Embodied Olympics - Fun | Standard | 56 | 27 | 20 | 20 | 16 | 13 | 26/56 |
| RoboDynamicExam | Standard | 10 | 4 | 3 | 4 | 1 | 2 | 4/10 |
| RoboCasa365 | Standard | 5 | 2 | 1 | 1 | 1 | 0 | 2/5 |
| RoboEngineer | Standard | 15 | 7 | 5 | 3 | 4 | 4 | 5/15 |
| RoboLab | Standard | 6 | 2 | 0 | 2 | 1 | 1 | 2/6 |
| BiGym 2.0 | Standard | 10 | 6 | 4 | 0 | 2 | 2 | 3/10 |
| MuJoCo Playground | Standard | 7 | 3 | 4 | 3 | 0 | 2 | 2/7 |
| RoboARC | Standard | 50 | 32 | 14 | 14 | 15 | 11 | 14/50 |
| DexHandExam | Standard | 23 | 15 | 13 | 9 | 2 | 6 | 6/23 |
| EmbodiedSWE | Standard | 12 | 8 | 5 | 5 | 6 | 4 | 3/12 |
| RoboWits | Standard | 6 | 3 | 2 | 3 | 4 | 2 | 1/6 |
| Embodied Olympics - Match | Standard | 13 | 10 | 6 | 6 | 3 | 6 | 2/13 |
| iSpatialGym | Standard | 8 | 7 | 4 | 5 | 3 | 3 | 1/8 |
| Embodied Safety Exam | Standard | 50 | 35 | 32 | 29 | 19 | 27 | 4/50 |
| ManiGuard | Standard | 7 | 6 | 3 | 5 | 7 | 4 | 0/7 |
Data as of . The stage labels, the controlled runs and the harness comparison in the findings are from the paper; the approach labels in Finding 4 come from a random sample of attempts (snapshot of ), drafted by a coding agent from the run logs and checked and finalized by the authors. Every clip shows the run it names, and clicking it opens a step-by-step view of that run. The leaderboard lists the same results for every suite.
04Findings
The paper reports four findings. Each one below gives its main evidence, a figure or a representative run, and a section you can expand with more cases and caveats. For the full analysis of where agents succeed and fail, see Where Do Today’s Agentic Robots Succeed and Fail?
The strongest agent, Claude Fable 5.2, solves of the Privileged tests but only of the Standard tests of the same tasks. In controlled runs with GPT-6 Astra, resets help more than the exact object state does. Stages that need precision become much harder in the Standard Setting.
- Every agent loses many of its Privileged successes in the Standard Setting (Figure 8). Even Claude Fable 5.2 fails of the tasks it solves in the Privileged Setting, and its success rate drops in suites with at least five tests in both settings.
- In controlled runs, we add the exact state of every object, unlimited resets with saved states, or both to GPT-6 Astra’s Standard interface. In these runs, resets raise its success rate from 29% to 53%, more than the state does, and with both it closes most of the gap to the Privileged Setting.
- Stages that need precision, such as a tight fit, a target pose or contact with a moving object, fail far more often in the Standard Setting than coarse stages do. At these stages resets help GPT-6 Astra, but the exact state alone does not.
Figure 8. The same task in two settings: each agent’s outcome on the core tasks that have both a Privileged and a Standard test. Every agent solves far more tasks only in the Privileged Setting than only in the Standard Setting. Each row is one agent on the 391 core tasks that have both tests; each segment is the share of tasks with that pair of outcomes.
The numbers
| Agent | Both | Only Privileged | Only Standard | Neither | Tasks |
|---|---|---|---|---|---|
| Claude Fable 5.2 | 175 | 137 | 9 | 70 | 391 |
| GPT-6 Astra | 116 | 192 | 4 | 79 | 391 |
| GPT-6.1 Sol | 110 | 176 | 5 | 100 | 391 |
| Claude Opus 5.5 | 86 | 147 | 5 | 153 | 391 |
| Gemini 4 Argon | 61 | 106 | 18 | 206 | 391 |
| Kimi 3.1 Pro | 25 | 123 | 9 | 234 | 391 |
| Gemini 3.8 Flash | 14 | 79 | 5 | 293 | 391 |
| Qwen3.8 Max | 7 | 30 | 3 | 351 | 391 |
| DeepSeek V4.1 Flash | 7 | 59 | 3 | 322 | 391 |
| GLM 5.3 Flash | 3 | 35 | 1 | 352 | 391 |
More cases and scope
In the Privileged Setting, an agent that breaks a rule can reset and try again; in the Standard Setting, one broken rule can be a point of no return. GPT-6 Astra on ManiGuard’s dusty transfer task shows both:
With resets and saved states, Claude Opus 5.5 solved the Bringing in wood task that it failed above, and Gemini 4 Argon solved a dribble slalom that it failed in the Standard Setting:
Scope
- Figure 8 covers all core tasks that have both a Privileged and a Standard test. The controlled runs are single runs of GPT-6 Astra in Codex on the default test of the same tasks (the Realtime test for 50 of them); the paper gives their details.
- A stage is precise when it needs a tight fit, a target pose or path, or contact with a moving object, and coarse when it needs only a region, a container, a grasp or the state of a mechanism.
This is usually the first stage of the task, and it also tends to be the same in the Privileged and Standard settings. Stages that change a material or move the robot’s body have the highest failure rates.
- We divide each task into stages, each a change in the world that the task requires: putting a tomato into a closed fridge has three, the fridge open, the tomato inside and the fridge closed again. For each failed attempt of the ten agents on 427 core tasks, coding agents (GPT-6.1 Sol and Claude Sonnet 5.5) label the stage at which it went wrong.
- For the strong agents, two failed attempts on the same test share their failure stage 55% of the time on average, against 30% if failure stages were random. The most common failure stage of a test is usually the first stage of its task.
- Leaving out tasks where it is the first stage in both settings, the most common failure stage is the same in the Privileged and the Standard Setting for 59% of the 88 tasks, against 9% by chance.
- Once reached, material stages such as wiping fail most often, at 56%, and body stages such as walking next, at 39%. Each category occurs mostly in particular suites and with particular robots, so these rates do not show that the category itself causes the failures.
More cases and scope
Three failures at stages that move the robot’s body or need a tight fit:
Scope
- A failed attempt has at most one failure stage: the stage of an irreversible event, such as a fall, or else of the earliest mistake that the agent did not correct. The labels come from a pipeline that combines labeling by coding agents with human spot checks while we developed the prompts.
The weaker agents are much more likely than the strong agents to fail before making the required contact. In the Standard Setting, the other four strong agents solve tests that Claude Fable 5.2 fails, and switching harnesses can change a model’s success rate by up to 7.5 percentage points.
- Counted per contact stage that they reach, the weaker agents fail without making the required contact at 9.0% to 18.5% of these stages, against 1.0% to 3.7% for the strong agents. The weaker agents also add few tests: of the Standard tests that any of them solves, the strong agents solve .
- In the Privileged Setting, Claude Fable 5.2 by itself solves of the tests that any agent solves. In the Standard Setting, no strong agent’s solved tests are a subset of another’s (Figure 9). The other four strong agents solve tests that Claude Fable 5.2 fails, and together the five solve of the Standard tests.
- The harness changes the result too. Run in Claude Code instead of its own Codex, GPT-6 Astra solves 7.5 percentage points fewer Standard tests, the largest change among the harnesses we compared.
Figure 9. Tests solved by one agent and by another (all the core Standard tests). No strong agent’s solved tests are a subset of another’s, and almost every test that a weaker agent solves is also solved by a strong agent. Each cell is the share of the row agent’s solved tests that the column agent also solves; the last column is the share that at least one agent above it on the leaderboard solves. The strong agents come first.
| Solved | Claude Fable 5.2 | GPT-6 Astra | GPT-6.1 Sol | Claude Opus 5.5 | Gemini 4 Argon | Kimi 3.1 Pro | Gemini 3.8 Flash | Qwen3.8 Max | DeepSeek V4.1 Flash | GLM 5.3 Flash | Any agent above | |||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Fable 5.2 | 226 | – | 52 | 52 | 41 | 42 | 17 | 10 | 6 | 5 | 5 | – | ||
| GPT-6 Astra | 157 | 75 | – | 62 | 43 | 48 | 21 | 13 | 10 | 7 | 5 | 75 | ||
| GPT-6.1 Sol | 147 | 80 | 67 | – | 52 | 50 | 23 | 13 | 10 | 8 | 6 | 90 | ||
| Claude Opus 5.5 | 114 | 81 | 60 | 68 | – | 55 | 30 | 18 | 13 | 9 | 8 | 95 | ||
| Gemini 4 Argon | 110 | 87 | 69 | 66 | 57 | – | 27 | 18 | 12 | 11 | 7 | 99 | ||
| Kimi 3.1 Pro | 46 | 85 | 72 | 74 | 74 | 65 | – | 30 | 28 | 22 | 15 | 98 | ||
| Gemini 3.8 Flash | 29 | 79 | 69 | 66 | 72 | 69 | 48 | – | 21 | 28 | 7 | 100 | ||
| Qwen3.8 Max | 17 | 82 | 88 | 82 | 88 | 76 | 76 | 35 | – | 47 | 35 | 100 | ||
| DeepSeek V4.1 Flash | 14 | 79 | 79 | 86 | 71 | 86 | 71 | 57 | 57 | – | 29 | 100 | ||
| GLM 5.3 Flash | 11 | 100 | 73 | 82 | 82 | 73 | 64 | 18 | 55 | 36 | – | 100 | ||
Shares in percent, of the 456 Standard tests. “Solved” is the number of tests that the row agent solves.
More cases and scope
Scope
- Each agent made one attempt per test, so part of these differences may come from the variation between runs. Three runs of GPT-6.1 Sol in the Standard Setting give a success rate of 30.8% ± 2.5 and differ by at most five points.
- Cross-provider runs of the harness comparison go through an adapter; the paper gives every model and harness pair.
In a sample of attempts, agents mostly control the robot directly in the Standard Setting and write programs in the Privileged and Realtime settings. Given a GPU and a training library, agents in the Standard and Realtime settings of a soccer task trained a policy, and eight of the nine wins used a trained policy.
- Where the simulation waits between requests, most sampled Standard attempts send one action chunk at a time and look at the images or readings before choosing the next. With the full state, and when the clock keeps running, most sampled attempts compute the motion in a program (Figure 10). Training a policy is rare: sampled attempts.
- The soccer match of Embodied Olympics - Match comes in two versions with the same frozen scene and opponent, and in both the agent controls two humanoids joint by joint. The GPU version adds a GPU, a reinforcement-learning library and GPU copies of the match for training. There most Standard and Realtime tests trained a policy, and every win came from one; on the CPU version, agents in these settings wrote gaits by hand and won once, in a penalty shoot-out. This one task cannot show which difference between the versions matters.
Figure 10. How agents control the robot: the approach of the sampled attempts, by setting. Agents mostly write a program in the Privileged and Realtime settings and mostly control the robot directly in the Standard Setting. Each segment is the share of a setting’s sampled attempts that used that approach (each attempt counted once, by the approach that produced its graded answer), as drafted by a coding agent from the run logs and checked and finalized by the authors.
Code to control
- Open-loop program computes the whole motion in advance (waypoints, key frames, inverse-kinematics solutions, a recorded sequence) and runs it without feedback
- Closed-loop program reads the state or the cameras at every step or segment and adjusts
- Parameter search finds a passing motion, or its parameters, by trying many episodes and resets or by running an optimizer (CEM, CMA, grid or random search)
Other approaches
- Direct control the agent sends commands one segment at a time and decides the next one after looking at images or readings
- Train a policy to control first trains a control policy, such as a neural network or a reinforcement-learning policy, that then drives the robot
- Mixed switched approach midway, or used two approaches equally
- No control plan only explored, or never ran anything
A dark mark separates the approaches; the three shades of blue are the three kinds of program. Sampled attempts: 2,622 analyzed in all (snapshot of 2026-10-18 to 2026-11-17, all ten agents).
The numbers
| Setting | Open-loop program | Closed-loop program | Parameter search | Direct control | Train a policy to control | Mixed | No control plan | Sampled |
|---|---|---|---|---|---|---|---|---|
| Privileged | 209 | 531 | 233 | 114 | 2 | 21 | 72 | 1182 |
| Standard | 113 | 203 | 103 | 755 | 9 | 68 | 12 | 1263 |
| Realtime | 29 | 131 | 7 | 1 | 4 | 4 | 1 | 177 |
More cases and scope
When the setting changes, so does the approach. GPT-6 Astra on RoboTwin:
In the Realtime Setting, the agent writes a program to play and lets it run:
Scope
- The approach counts are from a random sample of attempts (snapshot of ), drafted by a coding agent from the run logs and checked and finalized by the authors. The soccer counts include the runs of the harness comparison, so some models appear more than once.
05Future work
Running each core test several times will separate failures that an agent repeats from failures of a single run, and it will give confidence intervals for the comparisons between agents and between settings. We will also add more Realtime tasks, since the Realtime Setting has the largest share of tests that no agent solves.
Because the Standard Setting gives the agent only the robot’s sensor readings and control interface, the same interface can be used on physical hardware once there is a way to grade the outcome. In a first demonstration, two agents played a buzz wire game with a physical robot arm, and we plan to evaluate agents on real robots systematically.
Anyone can take part. A single command, harbor run -d embodied-first-exam -a <harness> -m <model>, evaluates a new model or harness on the core set; the code is in the exam repository. New tasks and whole suites are welcome (see how to contribute), every run can be replayed step by step on the runs page, and labs with real robots can reach us on Discord.
Limitations
Each agent makes a single attempt per test, the stage-level labels come from coding agents, and all results except the buzz-wire demonstration are in simulation. Each result belongs to a model in a particular harness, so the way the harness exposes tools, manages the context and ends a session is part of what we measure. The core set keeps the challenging tasks selected in trial runs, so its success rates are lower than those of the full exam, by an amount that differs from suite to suite.
References
- C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press and K. Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024.
- M. A. Merrill, A. G. Shaw, et al. Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces ICLR 2026.
- A. Ouyang, S. Guo, S. Arora, A. L. Zhang, W. Hu, C. Ré and A. Mirhoseini. KernelBench: Can LLMs Write Efficient GPU Kernels? ICML 2025.
- A. Brohan et al. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control CoRL 2023.
- M. J. Kim et al. OpenVLA: An Open-Source Vision-Language-Action Model CoRL 2024.
- K. Black et al. π0: A Vision-Language-Action Flow Model for General Robot Control arXiv 2024.
- J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence and A. Zeng. Code as Policies: Language Model Programs for Embodied Control ICRA 2023.
- C. Li, R. Zhang, et al. BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation arXiv 2024.
- J. X. Yang, R. Dagli, et al. RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies arXiv 2026.
- S. Nasiriany, A. Maddukuri, et al. RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots RSS 2024.
- S. Nasiriany, S. Nasiriany, et al. RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robots ICLR 2026.
- NVIDIA, J. Bjorck, et al. GR00T N1: An Open Foundation Model for Generalist Humanoid Robots arXiv 2025.
- R. McLean, E. Chatzaroulas, et al. Meta-World+: An Improved, Standardized, RL Benchmark NeurIPS 2025 Datasets and Benchmarks.
- S. Zhang, Z. Xu, et al. VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks arXiv 2024.
- C. Sferrazza, D. Huang, et al. HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulation arXiv 2024.
- Y. Kim, W. Pumacay, et al. MolmoSpaces: A Large-Scale Open Ecosystem for Robot Navigation and Manipulation arXiv 2026.
- Y. Huang, B. Li, et al. KinDER: A Physical Reasoning Benchmark for Robot Learning and Planning RSS 2026.
- T. Chen, Z. Chen, et al. RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation ICML 2026.
- K. Zakka, B. Tabanpour, et al. MuJoCo Playground arXiv 2025.
- K. Kedia, T. G. W. Lum, et al. SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation arXiv 2026.
- H. Wang, W. Zhao, et al. DexJoCo: A Benchmark and Toolkit for Task-Oriented Dexterous Manipulation on MuJoCo arXiv 2026.
- V. Caggiano, H. Wang, et al. MyoSuite -- A contact-rich simulation suite for musculoskeletal motor control arXiv 2022.
- T. Chen, Y. Chen, et al. RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies arXiv 2026.
- Z. Zhang, Z. Zhu, et al. BiGym 2.0: Benchmarking Learned and Agent-Developed Policies for Humanoid Household Manipulation arXiv 2026.
- ARISE Initiative. YAMLab GitHub repository, 2026.
- Y. Zhang, Z. Jin, et al. LIBERO-MAX: Do Robot Policies Adapt When the World Changes? arXiv 2026.
- Y. Wang, D. Lee, et al. AM-Bench: A Modular Simulation Suite and Benchmark for Aerial Manipulation Policy Learning arXiv 2026.
- C. Lin, H. Zhang, et al. RoboWits: Unexpected Challenges for Robotic Creative Problem Solving arXiv 2026.
- Y. Peng, P. Wang, et al. MANIGUARD: A Benchmark and Data Suite for Specification-Grounded Safety Evaluation and Improvement of Robotic Manipulation arXiv 2026.
- Z. Lan, Y. Jiang, et al. AutoBio: A Simulation and Benchmark for Robotic Automation in Digital Biology Laboratory arXiv 2025.
- R. Li, Z. Hu, et al. LabUtopia: High-Fidelity Simulation and Hierarchical Benchmark for Scientific Embodied Agents arXiv 2025.
- W. Zhang, K. Wang, et al. An Unexpected Robot Policy: Early Evaluations of GPT-6 Astra on RoboDojo and Beyond arXiv 2026.
Data and citation
The paper is on arXiv (PDF). The code is on GitHub. Tasks, results and agent runs are on this site (tasks, leaderboard, runs); videos are in the media dataset and the agent runs in the runs dataset.
@misc{embodiedfirstexam2026,
title = {Embodied First Exam},
author = {Kangrui Wang and Qineng Wang and Changhe Chen and Pingyue Zhang and Chi Gui and Feishi Wang and Canyu Chen and Wenli Xiao and Xiaoyu Huang and Tonghe Zhang and Linjie Li and Zihan Wang and Minyeong Kim and Yiyou Sun and Mingkai Deng and Jinyu Hou and Qi Zhu and Han Liu and Eric P. Xing and Huaxiu Yao and Zhiwen Fan and Yuejiang Liu and Dawn Song and Pieter Abbeel and Li Fei-Fei and Jiajun Wu and Yuke Zhu and Guanya Shi and Ruohan Zhang and Manling Li},
year = {2026},
eprint = {2611.12177},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.12177}
}