Blogs · Embodied First Exam

What is the first exam for coding agents to enter the physical world?

Embodied First Exam: An Agentic Robotics BenchmarkTesting Coding Agents for Robotics on 1,156 Tasks and 47 Robots

One line of code. 1,156 tasks. 35 suites, 47 robots.

  1. What robotic tasks can coding agents solve?

    One command connects a coding agent to 1,156 tasks in 35 suites, on robot arms, dexterous hands, humanoids, quadrupeds and musculoskeletal models. The agent chooses its own control approach: direct control, code to control, or train a policy to control.

  2. Under what physical-world constraints?

    The Privileged Setting gives the full simulator state and unlimited resets. The Standard Setting gives only the robot’s sensors and, by default, no resets, and its Realtime variant keeps the world running while the agent reasons.

  3. Where do they fail?

    We label the stage at which each failed attempt goes wrong and compare failures across agents, settings and stages. Every run is published with its full trace and a replay. The full analysis is in Where Do Today’s Agentic Robots Succeed and Fail?

RoboPaint · Oil, The Starry Nightreference solution

RoboARC · Untie a knotreference solution

RoboDynamicExam · Launch, catch, reloadreference solution

Embodied Olympics - Fun · Catch and shootGPT-6 Astra · Privileged

Embodied Olympics - Match · Fencingagent solution · wins 15:13

DexHandExam · Assemble a hamburgerreference solution

RoboEngineer · Weave between polesGPT-6 Astra · Standard

Embodied Safety Exam · Set the table on a place matreference solution

iSpatialGym · Match a goal imageGPT-6 Astra · Privileged

Figure 1. One task from each of the nine new suites. Four clips are coding agents’ own solutions, written with only a plain-language goal, a command-line shell and the robot’s interface: the Embodied Olympics - Fun catch and shoot, where the H1-2 humanoid catches a chest pass between its palms and shoots it through the hoop; an Embodied Olympics - Match épée bout between two Unitree G1 humanoids, which the agent wins 15:13 against a scripted fencer; a RoboEngineer run in which a Go2 quadruped weaves past four poles on a velocity schedule fixed in advance, with no position feedback; and an iSpatialGym task in which a mobile two-armed robot, given only a picture of the goal, makes the scene match it. The other clips are reference solutions, which show that these tasks can be solved. All clips are from the exam’s media dataset.

01Motivation

Coding agents have passed many exams in the digital world: they resolve issues in real repositories1, complete hard tasks in the terminal2 and write efficient GPU kernels3. But what is the first exam they must pass to enter the physical world? Robotics adds perception, contact and irreversible actions in a world that may keep moving while the agent thinks. Robots acquire new skills either by manual programming or by learning from data, as in vision-language-action models4–6. Coding agents suggest a third way (Figure 2), often called agentic robotics: they turn instructions into executable code7 and revise their plans through automated experimentation. We ask what robotic tasks general-purpose coding agents can solve, under what physical-world constraints, and where they fail, on tasks far beyond pick-and-place (Figure 3). The Embodied First Exam answers this with tasks and up to three evaluation settings per task, which separate what an agent can do with full simulator access from what it can do through the robot’s own sensors, without resets and on a running clock.

Today

Learn from data

A policy learns from many demonstrations. Good at the skills it was trained on, but a new task needs new data.

Today

Program it by hand

An engineer writes controllers and planners, tuned for one robot and one job.

This exam

A coding agent is the policy

Direct control, code to control or train a policy to control, from a plain-language instruction.

Figure 2. Three ways to give a robot a new skill. A policy learned from data needs new demonstrations for each new task, and manual programming builds a behavior for one robot and one task; in this exam, a coding agent takes the engineer’s place and itself becomes the policy.
Puzzles with spatial constraints

Untie a knot, take a ring off a horseshoe, free a hedgehog from its cage, make a scene match a picture of the goal.

RoboARC · iSpatialGym · RoboWits · KinDER
Sports and matches

Fence, play table tennis or soccer against another robot; catch a chest pass and shoot it through the hoop.

Embodied Olympics - Match · Embodied Olympics - Fun
Real-time control

The clock keeps running while the agent thinks: catch a thrown object, return a ball, react to an opponent.

RoboDynamicExam · Embodied Olympics - Match · Embodied Olympics - Fun · HumanoidBench · BiGym 2.0
Dynamic and whole-body control

Sprint, jump, dunk and backflip with a humanoid; walk and balance on four legs; control muscles instead of motors.

Embodied Olympics - Fun · RoboDynamicExam · HumanoidBench · MuJoCo Playground · MyoSuite
Sim-to-sim engineering

Tune a controller in a public simulator, deploy it to a private one whose dynamics differ, and revise it from the returned measurements.

RoboEngineer
Also in the exam

Dexterous hands and tools, long household activities in a full-size house, and safety: carry out harmless requests and refuse ones that would hurt someone.

DexHandExam · DexToolBench · BEHAVIOR-1K · RoboCasa365 · Embodied Safety Exam · ManiGuard
Figure 3. Kinds of task in the exam beyond pick-and-place, with suites that contain them. The tasks use robots of eight kinds: one arm (417 tasks), two arms (256 tasks), mobile robot (221 tasks), humanoid (142 tasks), dexterous hand (62 tasks), musculoskeletal (22 tasks), aerial robot (19 tasks), quadruped (17 tasks).

02Introduction

The exam is a robotics benchmark for coding agents: it tests them on 1,156 robot tasks under rules fixed in advance. Each task gives a goal in plain language and a frozen scene, and it can be taken in up to three settings, from full access to the simulator to the constraints of a real robot (Figure 4). The agent works in a sandboxed container whose only interface is a command-line shell: it reads files, writes and runs code, and reaches the robot through the interface of the setting. A verifier in its own sandbox grades what physically happened. Every task is packaged the same way, so one command runs different coding agents, each in its own harness, such as Codex or Claude Code. Most robotics benchmarks evaluate learned policies, like vision-language-action models. This exam evaluates the coding agent itself, on 47 robots of eight types.

Tasks and tests

The exam has 1,156 tasks and 2,288 tests in 35 suites, using 47 robots. A task is one frozen scene with one goal, and each pair of a task and a setting is a test. Nine of the suites are new (RoboPaint, RoboARC, RoboDynamicExam, Embodied Olympics - Fun, Embodied Olympics - Match, DexHandExam, RoboEngineer, Embodied Safety Exam and iSpatialGym). The other adapt existing benchmarks, and their scenes, robots and success criteria are what the exam is built on; we are grateful to their authors: BEHAVIOR-1K8, RoboLab9, RoboCasa10, RoboCasa36511, RoboCasa-GR112, MetaWorld+13, VLABench14, HumanoidBench15, MolmoSpaces16, KinDER17, RoboTwin 2.018, MuJoCo Playground19, DexToolBench20, DexJoCo21, MyoSuite22, RoboDojo23, BiGym 2.024, YAMLab25, LIBERO-MAX26, AM-Bench27, RoboWits28, ManiGuard29, AutoBio30, LabUtopia31, EmbodiedSWE and ABC Sim. You can browse the suites or the tasks.

A task in this exam is far larger than a question in a language benchmark. It asks for a complete behavior, such as assembling a piece of furniture or playing a whole match, that combines perception, planning, control and recovery from mistakes, and each attempt gets an engineer's budget: from to of agent time, depending on the suite, and a container with 2 to 16 CPU cores and 4 to 39 GiB of memory, with a GPU where the simulation backend or the task needs one. Tasks within a suite often test similar skills, and current agents already solve many of them.

We therefore report results on a core set of hard tasks ( tests). Experts among the authors and a coding agent selected the core set through trial runs, choosing tasks that are challenging and representative of their suite, and it keeps every task that the authors of a suite asked to keep. The other tasks stay in the exam and can be run with harbor run -d embodied-first-exam@all.

Three settings

A task can be run in up to three settings, and each one is a separate test, scored on its own. The rest of this post uses these three names exactly as defined here.

Privilegedinside the simulatorStandardreal-robot constraintsRealtimeon a running clock
Reachesthe simulator directly, in the agent’s containerthe robot only, through its APIthe robot only, through its API
Seesthe full state, every object poseRGB, depth and joints; no object posesas in Standard
Sendsa trajectory.npzaction chunks (T, dim)action chunks, often from a running program
Resetsunlimited0 by default; the task’s author can allow more0 by default; the task’s author can allow more
Clockplans offlinewaits for each commandkeeps running while the agent thinks
Gradedby a fresh replayon the live episodeon the live episode
Figure 4. The three settings. Most tasks have a Privileged and a Standard test. The Realtime variant is offered where timing matters, mostly for dynamic tasks with moving objects or opponents, and every core task has a Standard test. Every rule is on the page how a test is graded.

How one test runs

Figure 5 shows how one test runs, and the list below walks through it.

Harbor · one command
Question
goal in words Completely separate the two claw pieces and briefly hold both still.
frozen scene
  • rebuilt bit for bit
  • start state checked
  • same scene in every setting
Model
any model
Harness
CodexClaude Codeany harness
Interfacehow the code reaches the robot, per setting
Standardreal-time
reach
APIrobot API only
sees
RGBdepthjointsno object poses
sends
action chunks (T, dim)
How it runs underneath →
Privileged
reach
inside the simulator
sees
full stateresets ∞everything, at will
sends
trajectory.npzreplayed fresh
How it runs underneath →
Simulators47 robots
MuJoCoIsaac SimOmniGibsonSAPIENGenesis
recorded
Verifier
sandbox
live verdict
replay
10
steps · cost · video
Figure 5. How one test runs. Hover over a part for details, or click a control approach, a setting or the episode trajectory to see an example.
  1. Question. A test starts from a goal written in words, as you would explain the job to a person, with every convention the agent cannot guess (frames, units, the action layout, the time limit), and from a frozen scene. The scene is rebuilt bit for bit, its start state is checked before the agent begins, and all tests of a task start from the same scene.
  2. Model and harness. A model runs in its own provider’s harness, such as Codex or Claude Code, or in another provider’s harness through an adapter or a compatible API. The harness runs unchanged and receives the goal as its prompt. It works in a sandboxed container through a command-line shell and can reach only its model’s API and the package registries, with web search off; the Embodied Olympics - Fun tasks are the exception, with open network access.
  3. The coding agent is the policy. Whatever the agent writes turns what the robot senses into actions. It can control the robot directly, sending commands and choosing the next one after it sees the result; write code that controls the robot, such as a planner or a feedback controller; or train a policy, for example with reinforcement learning, that controls it. The exam leaves the choice to the agent.
  4. Interface. How the agent’s code reaches the robot depends on the setting. In the Privileged Setting the agent works inside the simulator; in the Standard Setting it reaches the robot only through the robot’s API, and the Realtime Setting is the same with a clock that never stops. In the Standard and Realtime settings the agent cannot reset the episode by default, although a task’s author can allow a fixed number of resets.
  5. Simulators. Each suite keeps its original simulation setup behind the same interface. The exam covers various simulators, including MuJoCo, Isaac Sim, OmniGibson (built on Isaac Sim), SAPIEN and Genesis, with different robot models running across them.
  6. Episode trajectory. Every episode is recorded step by step as a rollout: the full simulator state s, the robot’s observation o (camera images and joint readings) and the action a. What the agent itself can read depends on the setting: in the Privileged Setting it reads the full state s, while in the Standard and Realtime settings it sees only the observation o, as with a real robot. The verifier always works from the recorded state.
  7. Verifier. The verifier runs in its own sandbox with no network and sees only what was recorded. In the Standard and Realtime settings it grades the live episode; in the Privileged Setting it replays the trajectory. The test scores 1 or 0, and the run’s steps, cost and video are logged.

On RoboDojo, how fully the goal states the success check changes the results a great deal. We gave GPT-6 Astra the RoboDojo tasks in the Standard Setting with three versions of the goal text and kept everything else the same (Table 1). RoboDojo’s own one-sentence instructions leave out conditions that its success check tests, above all that both arms end near their start pose, which 47 of the 54 tasks require, so the agent often finishes the visible job and leaves an arm out of place. The text that RoboDojo gave GPT-6 Astra in its own evaluation32 adds a task recipe and a note about the start pose. With that text the agent solves almost as many tasks as with the exam’s goal, which states the success check in full, and most of the remaining differences come from how the manipulation turned out rather than from the wording. The exam’s goals therefore state the conditions they are graded on.

Goal textTasks solved
RoboDojo’s one-sentence instruction12 / 54
The same sentence, plus the task recipe (task description and scoring stages) and the note that both arms must return to their start pose: the text RoboDojo gave GPT-6 Astra in its own evaluation3233 / 54
The exam’s goal, which states RoboDojo’s success check in full37 / 54
Table 1. GPT-6 Astra on all 54 RoboDojo tasks in the Standard Setting with three versions of the goal text. Each version had one attempt per task, with the same scenes, step limits and success checks.

Three ways to control the robot

An agent can steer the robot itself (direct control), write the code that controls it (code to control), or train a policy that controls it (train a policy to control). The exam does not prescribe which, and agents mix them.

While the robot acts
Direct controlAgent⇄Robot

The agent is the controller: it sends a command, reads what the robot sensed, and chooses the next command, step by step, inside its own reasoning loop. Most attempts in the Standard Setting work this way. See it in a real run →

Code to controlCode→Robot

The agent writes a program that turns goals into actions through tools such as inverse kinematics, a motion planner or a feedback controller, and the program drives the robot on its own; between runs the agent reads the outcome and fixes the code. Most attempts in the Privileged and Realtime settings work this way. See it in a real run →

Train a policy to controlPolicy→Robot

The agent trains a policy, for example with reinforcement learning in simulation, and lets the trained policy drive the robot. The policy can be any network, not only a vision-language-action model. This is the rarest of the three. See it in a real run →

Keeping the grading honest

Reference solutions never enter the agent’s container, and we remove the original benchmarks’ solutions from the images. Doing nothing must score 0 on every task, and the reference solution must score 1 wherever one exists. Tasks without one, like most of BEHAVIOR-1K, rely on the original benchmark’s success checker. Before we add a suite, we tell an agent to try to pass without doing the task; if it succeeds anywhere, we hold the suite back until its verifier is fixed.

Two observations from this testing affect how the results should be read. First, models remember deleted solutions: Claude Opus 5.5 rewrote a deleted RoboTwin reference solution from memory, with 26 of its 28 lines identical to the original, and both Opus and GPT-6 Astra rebuilt a deleted HumanoidBench reward function with its original constants, so results on tasks built from public benchmarks may partly reflect memorization. Second, one harness offers tools that run on the provider's servers, outside the container and its network rules, and agents found and used them; these tools are turned off in the exam.

03Overall results

We evaluate coding agents on the core set. An agent is a model in a harness at high reasoning effort, and the harness is part of what we measure: Claude Fable 5.2 and Claude Opus 5.5 in Claude Code, GPT-6 Astra and GPT-6.1 Sol in Codex, Gemini 4 Argon in Antigravity CLI, Kimi 3.1 Pro and Qwen3.8 Max in Claude Code, and three Flash-class models, Gemini 3.8 Flash in Antigravity CLI, GLM 5.3 Flash in Claude Code and DeepSeek V4.1 Flash in DeepSeek Harness. Each agent makes one attempt per test under the same limits, and a test is solved when the verifier passes it. We report each setting on its own, Standard first, as on the leaderboard.

Figure 6. Success rate of each agent on the core set in each setting. Claude Fable 5.2 leads in every setting, and every agent solves by far its largest share of tests in the Privileged Setting. Each bar is the share of a setting’s core tests that one agent solved (the number on top is the percentage).

Claude Fable 5.2GPT-6 AstraGPT-6.1 SolClaude Opus 5.5Gemini 4 ArgonKimi 3.1 ProGemini 3.8 FlashQwen3.8 MaxDeepSeek V4.1 FlashGLM 5.3 Flash
0%20%40%60%80%Claude Fable 5.2 · Standard: 226 of 456 (50%)50GPT-6 Astra · Standard: 157 of 456 (34%)34GPT-6.1 Sol · Standard: 147 of 456 (32%)32Claude Opus 5.5 · Standard: 114 of 456 (25%)25Gemini 4 Argon · Standard: 110 of 456 (24%)24Kimi 3.1 Pro · Standard: 46 of 456 (10%)10Gemini 3.8 Flash · Standard: 29 of 456 (6%)6Qwen3.8 Max · Standard: 17 of 456 (4%)4DeepSeek V4.1 Flash · Standard: 14 of 456 (3%)3GLM 5.3 Flash · Standard: 11 of 456 (2%)Standard456 testsClaude Fable 5.2 · Privileged: 312 of 391 (80%)80GPT-6 Astra · Privileged: 308 of 391 (79%)79GPT-6.1 Sol · Privileged: 286 of 391 (73%)73Claude Opus 5.5 · Privileged: 233 of 391 (60%)60Gemini 4 Argon · Privileged: 167 of 391 (43%)43Kimi 3.1 Pro · Privileged: 148 of 391 (38%)38Gemini 3.8 Flash · Privileged: 93 of 391 (24%)24Qwen3.8 Max · Privileged: 37 of 391 (9%)9DeepSeek V4.1 Flash · Privileged: 66 of 391 (17%)17GLM 5.3 Flash · Privileged: 38 of 391 (10%)10Privileged391 testsClaude Fable 5.2 · Realtime: 22 of 54 (41%)41GPT-6 Astra · Realtime: 17 of 54 (31%)31GPT-6.1 Sol · Realtime: 13 of 54 (24%)24Claude Opus 5.5 · Realtime: 4 of 54 (7%)7Gemini 4 Argon · Realtime: 5 of 54 (9%)9Kimi 3.1 Pro · Realtime: 4 of 54 (7%)7Gemini 3.8 Flash · Realtime: 1 of 54 (2%)Qwen3.8 Max · Realtime: 1 of 54 (2%)DeepSeek V4.1 Flash · Realtime: 1 of 54 (2%)GLM 5.3 Flash · Realtime: 0 of 54 (0%)Realtime54 tests
The numbers
AgentStandardPrivilegedRealtime
Claude Fable 5.250% · 226/45680% · 312/39141% · 22/54
GPT-6 Astra34% · 157/45679% · 308/39131% · 17/54
GPT-6.1 Sol32% · 147/45673% · 286/39124% · 13/54
Claude Opus 5.525% · 114/45660% · 233/3917% · 4/54
Gemini 4 Argon24% · 110/45643% · 167/3919% · 5/54
Kimi 3.1 Pro10% · 46/45638% · 148/3917% · 4/54
Gemini 3.8 Flash6% · 29/45624% · 93/3912% · 1/54
Qwen3.8 Max4% · 17/4569% · 37/3912% · 1/54
DeepSeek V4.1 Flash3% · 14/45617% · 66/3912% · 1/54
GLM 5.3 Flash2% · 11/45610% · 38/3910% · 0/54
Claude Fable 5.2 solves the most tests in every setting. GPT-6 Astra and GPT-6.1 Sol follow it in every setting, and Claude Opus 5.5 and Gemini 4 Argon come next in the Standard Setting. We call these five the strong agents and the other five the weaker agents. The Realtime Setting is harder still: on the tasks with both tests, of the Realtime tests remain unsolved by any agent, against of the Standard tests.

At least one strong agent solves of the Standard tests. Yet in suites, at least half of the Standard tests are solved by none of them, while in the Privileged Setting this holds (Figure 7). Averaged over the strong agents, the success rate is at least 70% in Privileged suites and .

Figure 7. Success rate of each strong agent, by suite (all the core tests of the setting). In the Privileged Setting, one suite has at least half its tests solved by none of the strong agents; in the Standard Setting, seven. Each dot is one agent’s share of a suite’s tests; the gray bar on the right is the share that none of the strong agents solves.

Claude Fable 5.2GPT-6 AstraGPT-6.1 SolClaude Opus 5.5Gemini 4 ArgonNone of the strong agents solves (right)

Privileged

None of the strong agents solves
HumanoidBenchhumanoid whole-body control4/8
below half
MyoSuitemusculoskeletal hand6/13
0255075100%

Standard

None of the strong agents solves
HumanoidBenchhumanoid whole-body control7/8
IKEA assemblytwo-arm precision assembly7/8
DexToolBenchdexterous hand + tools17/20
MyoSuitemusculoskeletal hand8/13
BEHAVIOR-1Klong household tasks36/71
RoboPaintbrush strokes11/22
RoboDojotwo-arm long sequences6/12
below half
Embodied Olympics - Funhumanoid sports26/56
0255075100%

Hardest first. Only suites with at least 5 tests in the setting. Each column lists the suites with at least half their tests solved by none of the strong agents, then the next one below the dashed line. The small label under each name is the kind of task. The IKEA assembly tasks of RoboARC count as their own suite.

The numbers
SuiteSettingTestsFable 5.2GPT-6 AstraGPT-6.1 SolOpus 5.5Gemini 4 ArgonNone of the strong agents solves
HumanoidBenchPrivileged8333224/8
MyoSuitePrivileged13667506/13
BEHAVIOR-1KPrivileged7136363822619/71
DexToolBenchPrivileged20101310014/20
iSpatialGymPrivileged8777541/8
Embodied Olympics - FunPrivileged5649494545325/56
RoboDojoPrivileged1211106331/12
RoboARCPrivileged5046413638251/50
DexHandExamPrivileged232222181680/23
RoboPaintPrivileged2220222011150/22
Embodied Olympics - MatchPrivileged1312111212120/13
EmbodiedSWEPrivileged121211111090/12
RoboDynamicExamPrivileged101010101050/10
BiGym 2.0Privileged1010109570/10
IKEA assemblyPrivileged8764520/8
MuJoCo PlaygroundPrivileged7777540/7
ManiGuardPrivileged7777770/7
RoboLabPrivileged6554310/6
RoboWitsPrivileged6555550/6
RoboCasa365Privileged5555430/5
HumanoidBenchStandard8010007/8
IKEA assemblyStandard8110007/8
DexToolBenchStandard201210017/20
MyoSuiteStandard13533318/13
BEHAVIOR-1KStandard712614135436/71
RoboPaintStandard222762211/22
RoboDojoStandard12630246/12
Embodied Olympics - FunStandard56272020161326/56
RoboDynamicExamStandard10434124/10
RoboCasa365Standard5211102/5
RoboEngineerStandard15753445/15
RoboLabStandard6202112/6
BiGym 2.0Standard10640223/10
MuJoCo PlaygroundStandard7343022/7
RoboARCStandard50321414151114/50
DexHandExamStandard2315139266/23
EmbodiedSWEStandard12855643/12
RoboWitsStandard6323421/6
Embodied Olympics - MatchStandard131066362/13
iSpatialGymStandard8745331/8
Embodied Safety ExamStandard5035322919274/50
ManiGuardStandard7635740/7

Data as of . The stage labels, the controlled runs and the harness comparison in the findings are from the paper; the approach labels in Finding 4 come from a random sample of attempts (snapshot of ), drafted by a coding agent from the run logs and checked and finalized by the authors. Every clip shows the run it names, and clicking it opens a step-by-step view of that run. The leaderboard lists the same results for every suite.

04Findings

The paper reports four findings. Each one below gives its main evidence, a figure or a representative run, and a section you can expand with more cases and caveats. For the full analysis of where agents succeed and fail, see Where Do Today’s Agentic Robots Succeed and Fail?

The strongest agent, Claude Fable 5.2, solves of the Privileged tests but only of the Standard tests of the same tasks. In controlled runs with GPT-6 Astra, resets help more than the exact object state does. Stages that need precision become much harder in the Standard Setting.

  • Every agent loses many of its Privileged successes in the Standard Setting (Figure 8). Even Claude Fable 5.2 fails of the tasks it solves in the Privileged Setting, and its success rate drops in suites with at least five tests in both settings.
  • In controlled runs, we add the exact state of every object, unlimited resets with saved states, or both to GPT-6 Astra’s Standard interface. In these runs, resets raise its success rate from 29% to 53%, more than the state does, and with both it closes most of the gap to the Privileged Setting.
  • Stages that need precision, such as a tight fit, a target pose or contact with a moving object, fail far more often in the Standard Setting than coarse stages do. At these stages resets help GPT-6 Astra, but the exact state alone does not.

Figure 8. The same task in two settings: each agent’s outcome on the core tasks that have both a Privileged and a Standard test. Every agent solves far more tasks only in the Privileged Setting than only in the Standard Setting. Each row is one agent on the 391 core tasks that have both tests; each segment is the share of tasks with that pair of outcomes.

Solved in bothOnly in PrivilegedOnly in StandardIn neither
Claude Fable 5.217513770391
GPT-6 Astra11619279391
GPT-6.1 Sol110176100391
Claude Opus 5.586147153391
Gemini 4 Argon61106206391
Kimi 3.1 Pro123234391
Gemini 3.8 Flash79293391
Qwen3.8 Max351391
DeepSeek V4.1 Flash59322391
GLM 5.3 Flash352391
0%50%100%tasks
The numbers
AgentBothOnly PrivilegedOnly StandardNeitherTasks
Claude Fable 5.2175137970391
GPT-6 Astra116192479391
GPT-6.1 Sol1101765100391
Claude Opus 5.5861475153391
Gemini 4 Argon6110618206391
Kimi 3.1 Pro251239234391
Gemini 3.8 Flash14795293391
Qwen3.8 Max7303351391
DeepSeek V4.1 Flash7593322391
GLM 5.3 Flash3351352391
More cases and scope

In the Privileged Setting, an agent that breaks a rule can reset and try again; in the Standard Setting, one broken rule can be a point of no return. GPT-6 Astra on ManiGuard’s dusty transfer task shows both:

With resets and saved states, Claude Opus 5.5 solved the Bringing in wood task that it failed above, and Gemini 4 Argon solved a dribble slalom that it failed in the Standard Setting:

Scope

  • Figure 8 covers all core tasks that have both a Privileged and a Standard test. The controlled runs are single runs of GPT-6 Astra in Codex on the default test of the same tasks (the Realtime test for 50 of them); the paper gives their details.
  • A stage is precise when it needs a tight fit, a target pose or path, or contact with a moving object, and coarse when it needs only a region, a container, a grasp or the state of a mechanism.

This is usually the first stage of the task, and it also tends to be the same in the Privileged and Standard settings. Stages that change a material or move the robot’s body have the highest failure rates.

  • We divide each task into stages, each a change in the world that the task requires: putting a tomato into a closed fridge has three, the fridge open, the tomato inside and the fridge closed again. For each failed attempt of the ten agents on 427 core tasks, coding agents (GPT-6.1 Sol and Claude Sonnet 5.5) label the stage at which it went wrong.
  • For the strong agents, two failed attempts on the same test share their failure stage 55% of the time on average, against 30% if failure stages were random. The most common failure stage of a test is usually the first stage of its task.
  • Leaving out tasks where it is the first stage in both settings, the most common failure stage is the same in the Privileged and the Standard Setting for 59% of the 88 tasks, against 9% by chance.
  • Once reached, material stages such as wiping fail most often, at 56%, and body stages such as walking next, at 39%. Each category occurs mostly in particular suites and with particular robots, so these rates do not show that the category itself causes the failures.
More cases and scope

Three failures at stages that move the robot’s body or need a tight fit:

Scope

  • A failed attempt has at most one failure stage: the stage of an irreversible event, such as a fall, or else of the earliest mistake that the agent did not correct. The labels come from a pipeline that combines labeling by coding agents with human spot checks while we developed the prompts.

The weaker agents are much more likely than the strong agents to fail before making the required contact. In the Standard Setting, the other four strong agents solve tests that Claude Fable 5.2 fails, and switching harnesses can change a model’s success rate by up to 7.5 percentage points.

  • Counted per contact stage that they reach, the weaker agents fail without making the required contact at 9.0% to 18.5% of these stages, against 1.0% to 3.7% for the strong agents. The weaker agents also add few tests: of the Standard tests that any of them solves, the strong agents solve .
  • In the Privileged Setting, Claude Fable 5.2 by itself solves of the tests that any agent solves. In the Standard Setting, no strong agent’s solved tests are a subset of another’s (Figure 9). The other four strong agents solve tests that Claude Fable 5.2 fails, and together the five solve of the Standard tests.
  • The harness changes the result too. Run in Claude Code instead of its own Codex, GPT-6 Astra solves 7.5 percentage points fewer Standard tests, the largest change among the harnesses we compared.

Figure 9. Tests solved by one agent and by another (all the core Standard tests). No strong agent’s solved tests are a subset of another’s, and almost every test that a weaker agent solves is also solved by a strong agent. Each cell is the share of the row agent’s solved tests that the column agent also solves; the last column is the share that at least one agent above it on the leaderboard solves. The strong agents come first.

SolvedClaude Fable 5.2GPT-6 AstraGPT-6.1 SolClaude Opus 5.5Gemini 4 ArgonKimi 3.1 ProGemini 3.8 FlashQwen3.8 MaxDeepSeek V4.1 FlashGLM 5.3 FlashAny agent above
Claude Fable 5.2226–525241421710655–
GPT-6 Astra15775–6243482113107575
GPT-6.1 Sol1478067–52502313108690
Claude Opus 5.5114816068–553018139895
Gemini 4 Argon11087696657–27181211799
Kimi 3.1 Pro468572747465–3028221598
Gemini 3.8 Flash29796966726948–21287100
Qwen3.8 Max1782888288767635–4735100
DeepSeek V4.1 Flash147979867186715757–29100
GLM 5.3 Flash111007382827364185536–100

Shares in percent, of the 456 Standard tests. “Solved” is the number of tests that the row agent solves.

More cases and scope

Scope

  • Each agent made one attempt per test, so part of these differences may come from the variation between runs. Three runs of GPT-6.1 Sol in the Standard Setting give a success rate of 30.8% ± 2.5 and differ by at most five points.
  • Cross-provider runs of the harness comparison go through an adapter; the paper gives every model and harness pair.

In a sample of attempts, agents mostly control the robot directly in the Standard Setting and write programs in the Privileged and Realtime settings. Given a GPU and a training library, agents in the Standard and Realtime settings of a soccer task trained a policy, and eight of the nine wins used a trained policy.

  • Where the simulation waits between requests, most sampled Standard attempts send one action chunk at a time and look at the images or readings before choosing the next. With the full state, and when the clock keeps running, most sampled attempts compute the motion in a program (Figure 10). Training a policy is rare: sampled attempts.
  • The soccer match of Embodied Olympics - Match comes in two versions with the same frozen scene and opponent, and in both the agent controls two humanoids joint by joint. The GPU version adds a GPU, a reinforcement-learning library and GPU copies of the match for training. There most Standard and Realtime tests trained a policy, and every win came from one; on the CPU version, agents in these settings wrote gaits by hand and won once, in a penalty shoot-out. This one task cannot show which difference between the versions matters.

Figure 10. How agents control the robot: the approach of the sampled attempts, by setting. Agents mostly write a program in the Privileged and Realtime settings and mostly control the robot directly in the Standard Setting. Each segment is the share of a setting’s sampled attempts that used that approach (each attempt counted once, by the approach that produced its graded answer), as drafted by a coding agent from the run logs and checked and finalized by the authors.

Code to control

  • Open-loop program computes the whole motion in advance (waypoints, key frames, inverse-kinematics solutions, a recorded sequence) and runs it without feedback
  • Closed-loop program reads the state or the cameras at every step or segment and adjusts
  • Parameter search finds a passing motion, or its parameters, by trying many episodes and resets or by running an optimizer (CEM, CMA, grid or random search)

Other approaches

  • Direct control the agent sends commands one segment at a time and decides the next one after looking at images or readings
  • Train a policy to control first trains a control policy, such as a neural network or a reinforcement-learning policy, that then drives the robot
  • Mixed switched approach midway, or used two approaches equally
  • No control plan only explored, or never ran anything
PrivilegedCode to control 973/1182Code to control 973/1182 · Direct control 114/11821,182
StandardCode to control 419/1263Direct control 755/1263Code to control 419/1263 · Direct control 755/12631,263
RealtimeCode to control 167/177Code to control 167/177 · Direct control 1/177177
0%50%100%attempts

A dark mark separates the approaches; the three shades of blue are the three kinds of program. Sampled attempts: 2,622 analyzed in all (snapshot of 2026-10-18 to 2026-11-17, all ten agents).

The numbers
SettingOpen-loop programClosed-loop programParameter searchDirect controlTrain a policy to controlMixedNo control planSampled
Privileged209531233114221721182
Standard113203103755968121263
Realtime2913171441177
More cases and scope

When the setting changes, so does the approach. GPT-6 Astra on RoboTwin:

In the Realtime Setting, the agent writes a program to play and lets it run:

Scope

  • The approach counts are from a random sample of attempts (snapshot of ), drafted by a coding agent from the run logs and checked and finalized by the authors. The soccer counts include the runs of the harness comparison, so some models appear more than once.

05Future work

Running each core test several times will separate failures that an agent repeats from failures of a single run, and it will give confidence intervals for the comparisons between agents and between settings. We will also add more Realtime tasks, since the Realtime Setting has the largest share of tests that no agent solves.

Because the Standard Setting gives the agent only the robot’s sensor readings and control interface, the same interface can be used on physical hardware once there is a way to grade the outcome. In a first demonstration, two agents played a buzz wire game with a physical robot arm, and we plan to evaluate agents on real robots systematically.

Anyone can take part. A single command, harbor run -d embodied-first-exam -a <harness> -m <model>, evaluates a new model or harness on the core set; the code is in the exam repository. New tasks and whole suites are welcome (see how to contribute), every run can be replayed step by step on the runs page, and labs with real robots can reach us on Discord.

Limitations

Each agent makes a single attempt per test, the stage-level labels come from coding agents, and all results except the buzz-wire demonstration are in simulation. Each result belongs to a model in a particular harness, so the way the harness exposes tools, manages the context and ends a session is part of what we measure. The core set keeps the challenging tasks selected in trial runs, so its success rates are lower than those of the full exam, by an amount that differs from suite to suite.

References

  1. C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press and K. Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024.
  2. M. A. Merrill, A. G. Shaw, et al. Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces ICLR 2026.
  3. A. Ouyang, S. Guo, S. Arora, A. L. Zhang, W. Hu, C. Ré and A. Mirhoseini. KernelBench: Can LLMs Write Efficient GPU Kernels? ICML 2025.
  4. A. Brohan et al. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control CoRL 2023.
  5. M. J. Kim et al. OpenVLA: An Open-Source Vision-Language-Action Model CoRL 2024.
  6. K. Black et al. π0: A Vision-Language-Action Flow Model for General Robot Control arXiv 2024.
  7. J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence and A. Zeng. Code as Policies: Language Model Programs for Embodied Control ICRA 2023.
  8. C. Li, R. Zhang, et al. BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation arXiv 2024.
  9. J. X. Yang, R. Dagli, et al. RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies arXiv 2026.
  10. S. Nasiriany, A. Maddukuri, et al. RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots RSS 2024.
  11. S. Nasiriany, S. Nasiriany, et al. RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robots ICLR 2026.
  12. NVIDIA, J. Bjorck, et al. GR00T N1: An Open Foundation Model for Generalist Humanoid Robots arXiv 2025.
  13. R. McLean, E. Chatzaroulas, et al. Meta-World+: An Improved, Standardized, RL Benchmark NeurIPS 2025 Datasets and Benchmarks.
  14. S. Zhang, Z. Xu, et al. VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks arXiv 2024.
  15. C. Sferrazza, D. Huang, et al. HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulation arXiv 2024.
  16. Y. Kim, W. Pumacay, et al. MolmoSpaces: A Large-Scale Open Ecosystem for Robot Navigation and Manipulation arXiv 2026.
  17. Y. Huang, B. Li, et al. KinDER: A Physical Reasoning Benchmark for Robot Learning and Planning RSS 2026.
  18. T. Chen, Z. Chen, et al. RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation ICML 2026.
  19. K. Zakka, B. Tabanpour, et al. MuJoCo Playground arXiv 2025.
  20. K. Kedia, T. G. W. Lum, et al. SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation arXiv 2026.
  21. H. Wang, W. Zhao, et al. DexJoCo: A Benchmark and Toolkit for Task-Oriented Dexterous Manipulation on MuJoCo arXiv 2026.
  22. V. Caggiano, H. Wang, et al. MyoSuite -- A contact-rich simulation suite for musculoskeletal motor control arXiv 2022.
  23. T. Chen, Y. Chen, et al. RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies arXiv 2026.
  24. Z. Zhang, Z. Zhu, et al. BiGym 2.0: Benchmarking Learned and Agent-Developed Policies for Humanoid Household Manipulation arXiv 2026.
  25. ARISE Initiative. YAMLab GitHub repository, 2026.
  26. Y. Zhang, Z. Jin, et al. LIBERO-MAX: Do Robot Policies Adapt When the World Changes? arXiv 2026.
  27. Y. Wang, D. Lee, et al. AM-Bench: A Modular Simulation Suite and Benchmark for Aerial Manipulation Policy Learning arXiv 2026.
  28. C. Lin, H. Zhang, et al. RoboWits: Unexpected Challenges for Robotic Creative Problem Solving arXiv 2026.
  29. Y. Peng, P. Wang, et al. MANIGUARD: A Benchmark and Data Suite for Specification-Grounded Safety Evaluation and Improvement of Robotic Manipulation arXiv 2026.
  30. Z. Lan, Y. Jiang, et al. AutoBio: A Simulation and Benchmark for Robotic Automation in Digital Biology Laboratory arXiv 2025.
  31. R. Li, Z. Hu, et al. LabUtopia: High-Fidelity Simulation and Hierarchical Benchmark for Scientific Embodied Agents arXiv 2025.
  32. W. Zhang, K. Wang, et al. An Unexpected Robot Policy: Early Evaluations of GPT-6 Astra on RoboDojo and Beyond arXiv 2026.

Data and citation

The paper is on arXiv (PDF). The code is on GitHub. Tasks, results and agent runs are on this site (tasks, leaderboard, runs); videos are in the media dataset and the agent runs in the runs dataset.

@misc{embodiedfirstexam2026,
  title        = {Embodied First Exam},
  author       = {Kangrui Wang and Qineng Wang and Changhe Chen and Pingyue Zhang and Chi Gui and Feishi Wang and Canyu Chen and Wenli Xiao and Xiaoyu Huang and Tonghe Zhang and Linjie Li and Zihan Wang and Minyeong Kim and Yiyou Sun and Mingkai Deng and Jinyu Hou and Qi Zhu and Han Liu and Eric P. Xing and Huaxiu Yao and Zhiwen Fan and Yuejiang Liu and Dawn Song and Pieter Abbeel and Li Fei-Fei and Jiajun Wu and Yuke Zhu and Guanya Shi and Ruohan Zhang and Manling Li},
  year         = {2026},
  eprint       = {2611.12177},
  archivePrefix = {arXiv},
  url          = {https://arxiv.org/abs/2611.12177}
}