Embodied First Exam

One line of code. tasks. benchmarks, robots.

1. What can it connect, and what can it do?

   One command connects a coding agent to tasks ( tests) from benchmark suites, on robot arms, dexterous hands, humanoids, quadrupeds and musculoskeletal models.

2. When it fails, is it thinking or moving?

   Every run is published with its full trace and a replay, so you can see where it went wrong. A hand-checked classification of the failures is planned (§05).

3. How does a coding agent become an effective robot controller?

   We compare the three ways agents control a robot (write a program, send commands step by step, or train a model) under three settings, the hardest of which keeps the world moving while the agent thinks.

01Motivation

Code agents now handle a wide range of software work. They resolve issues in real repositories¹, complete hard tasks in the terminal² and write efficient GPU kernels³. All of these exams, however, take place in the digital world. Robots remain largely outside the reach of code agents: a robot behavior today comes either from a policy trained on many demonstrations, such as a vision-language-action model^4–6, or from a controller that an engineer writes, tunes and debugs for one robot and one task.

The second route is, to a large extent, software engineering. The engineer reads the interface of the robot, writes a controller, runs it, observes what happens and revises the code. Prior work has shown that language models can write short robot programs from instructions⁷; a code agent can go further and carry out the whole loop. If it succeeds, any robot that exposes a programming interface could acquire a new behavior from a plain-language instruction, with no new demonstrations and no engineer in the loop (Figure 2).

Whether code agents can do this remains an open question, and answering it requires more than simply testing them on pick-and-place tasks. Most robot benchmarks ask an arm to move objects on a table, whereas a robotics engineer also faces dynamic interaction, whole-body control of humanoids and quadrupeds, dexterous manipulation with tools, physical puzzles and long household activities (Figure 3).

An answer also needs a common interface that runs any agent on any task, and a verifier that judges what physically happened in the simulator rather than what the agent reports. The Embodied First Exam is built to meet these requirements, and it serves as a first exam for code agents that are about to enter the physical world.

02Introduction

Every task in the exam is packaged the same way and runs through Harbor, so one command can evaluate any agent on any task. For each benchmark we integrate its simulation environment, including scenes and success checkers, and the exam turns them into tests. The agent takes a test from inside a sandboxed container, where its only interface is a command-line shell: it reads files, writes and runs code, and calls the robot through the interface of the setting. The figure shows how one test runs, and the list below walks through it.

1. Question. A test starts from a goal written in words, as you would explain the job to a person, with every convention the agent cannot guess (frames, units, the action layout, the time limit), and from a frozen scene. The scene is rebuilt bit for bit, its start state is checked before the agent begins, and both modes of a task start from the same scene.
2. Model and harness. Any model can run in any harness, such as Codex or Claude Code. The harness runs unchanged and receives the goal as its prompt. It works in a sandboxed container through a command-line shell and can reach only its model’s API and the package registries, with web search off; the Embodied Olympics - Fun tasks are the exception, with open network access.
3. The coding agent is the policy. Whatever the agent writes turns what the robot senses into actions. It can control the robot directly, sending commands and choosing the next one after it sees the result; write code that controls the robot, such as a planner or a feedback controller; or train a model, such as a reinforcement-learning policy, that controls it. The exam leaves the choice to the agent.
4. Interface. How the agent’s code reaches the robot depends on the setting. In privileged mode the agent works inside the simulator; in standard mode it reaches the robot only through the robot’s API, and standard-realtime is the same with a clock that never stops. In both standard settings the agent cannot reset the episode by default, although a task’s author can allow a fixed number of resets. The three settings are defined below the list.
5. Simulators. Each suite keeps its original simulation setup behind the same interface. The exam covers various simulators, including MuJoCo, Isaac Sim, OmniGibson (built on Isaac Sim), SAPIEN and Genesis, with different robot models running across them.
6. Episode trajectory. Every episode is recorded step by step as a rollout: the full simulator state s, the robot’s observation o (camera images and joint readings) and the action a. What the agent itself can read depends on the setting: in privileged mode it reads the full state s, while in the two standard settings it sees only the observation o, as with a real robot. The verifier always works from the recorded state.
7. Verifier. The verifier runs in its own sandbox with no network and sees only what was recorded. In standard mode it gives a verdict on the live episode; in privileged mode it replays the trajectory. The test scores 1 or 0, and the run’s steps, cost and video are logged.

Three ways to control the robot

Three settings

A task can be run in up to three settings, and each one is a separate test, scored on its own. The rest of this post uses these three names exactly as defined here.

Tasks and tests

The exam has tasks and tests from benchmark suites, using robots. A task is one fixed scene with one goal, and each of its settings is a test. Most suites are ports of widely used benchmarks (BEHAVIOR-1K, RoboCasa, LIBERO, RoboTwin 2.0, MuJoCo Playground, MetaWorld+ and others); we built the other nine for this exam (RoboPaint, RoboARC, RoboDynamicExam, Embodied Olympics - Fun, Embodied Olympics - Match, DexHandExam, RoboEngineer, Embodied Safety Exam and iSpatialGym). You can browse the suites or the tasks.

A task in this exam is far larger than a question in a language benchmark. It asks for a complete behavior, such as assembling a piece of furniture or playing a whole match, that combines perception, planning, control and recovery from mistakes, and each attempt gets an engineer's budget: from to of agent time, depending on the suite, and a container with 2 to 16 CPU cores and 6 to 40 GB of memory, with a GPU where the simulation backend or the task needs one. Many tasks in the same benchmark also test nearly the same skills, so a few hard tasks cover what a benchmark measures.

We therefore report results on a core set of tasks ( tests), the harder part of the exam. We use human judgment and an initial run with GPT-6 Luna to filter out easier tasks, and we keep every task that a benchmark’s authors asked us to keep. The other tasks stay in the exam and can be run with harbor run -d embodied-first-exam@all.

Keeping the grading honest

Reference solutions never enter the agent’s container, and we remove the original benchmarks’ solutions from the images. Doing nothing must score 0 on every task, and the reference solution must score 1 wherever one exists (a full reference solution for tasks, a partial one for ). Tasks without one, like most of BEHAVIOR-1K, rely on the original benchmark’s success checker. Before we add a benchmark, we tell an agent to try to pass without doing the task; if it succeeds anywhere, we hold the benchmark back until its verifier is fixed.

Two observations from this testing affect how the results should be read. First, models remember deleted solutions: Claude Opus 5.5 rewrote a deleted RoboTwin reference solution from memory, with 26 of its 28 lines identical to the original, and both Opus and GPT-6 Astra rebuilt a deleted HumanoidBench reward function with its original constants, so results on tasks built from public benchmarks may partly reflect memorization. Second, one harness offered tools that ran on the provider's servers, outside the container and its network rules, and agents found and used them; these tools are now turned off, and the gateway in front of the model forwards only model calls.

03Overall results

We evaluate agents on the core set, each a model in a harness at high reasoning effort: GPT-6 Astra in Codex, Claude Opus 5.5 in Claude Code and Gemini 4 Argon in the Antigravity CLI, together with three Flash-class models, Gemini 3.8 Flash in the Antigravity CLI, GLM 5.3 Flash in Claude Code and DeepSeek V4.1 Flash in the DeepSeek harness. A test counts as solved when the verifier returns 1 on the latest finished attempt of the agent, and a test without a result counts against the agent. We report four views: Core, which counts each of the core tasks once in its default setting, and one view for each setting over all of its core tests ( privileged, standard and standard-realtime). The core set is hard by design: easier tasks were filtered out by human judgment and an initial run with GPT-6 Luna.

Data as of . The approach labels in Finding 5 come from a random sample of attempts (snapshot of ), labelled by an agent that read the logs of each run and not yet checked by hand. Every clip in this post shows the run it names, and clicking it opens a step-by-step view of that run. The leaderboard lists the same results for every suite.

04Findings

Click a finding to open it. Each one shows the main numbers, a figure and one representative run, followed by a section you can expand with more evidence, more cases and caveats.

05Future work

The exam currently reports whether an attempt passed, not why it failed. We plan to label every failed attempt with a single cause: the agent misread the task, the grasp missed, the robot went out of control, or the agent declared success before the goal was met. Each label will be checked by a person against the trace and the replay, so that statements such as the one in Finding 3 can rest on counted causes rather than on the suites in which failures occur.

Each agent currently makes one attempt per test. Averaged over hundreds of tests, this is enough to rank agents, but a single test can still go either way. Running every core test several times will separate failures that an agent repeats from failures that come from one unlucky run, and will put confidence intervals on the comparisons in Findings 1 and 2.

Most agents in this post pair a model with its maker's own harness: GPT-6 Astra runs in Codex, Claude Opus 5.5 in Claude Code and the Gemini models in the Antigravity CLI. The exam runs any model in any harness, so the same model can also be tested in several harnesses, for example an open-weight model in Claude Code and in the harness of its maker. Such comparisons would separate what the harness contributes, through its tools, its handling of context and its decision of when to stop, from what the model contributes, and we plan to add them to the results.

Only core tests run in the standard-realtime setting, yet this is the setting in which every agent does worst. We will add more tasks in which the world keeps moving, and study whether agents notice when an attempt goes wrong and recover from it when the simulator can neither pause nor reset.

Finally, the standard setting gives the agent only what a real robot provides: camera images, depth, joint readings and a control interface. The same tests can therefore run on physical hardware, and we plan to evaluate agents on real robots under the standard rules.

Anyone can take part. A single command, harbor run -d embodied-first-exam -a <harness> -m <model>, evaluates a new model or harness on the core set; the code is in the exam repository. New tasks and whole benchmarks are welcome (see how to contribute), every run can be replayed step by step on the runs page, and labs with real robots can reach us on Discord.

Limitations

Each result belongs to a model in a particular harness, so the way the harness uses tools, manages context and decides when to stop is part of what we measure. Every test has a single attempt, as discussed above. The core set leaves out tests that we judged easy and tests that an early screening run solved, so its success rates are lower than those of the full exam, by an amount that differs from suite to suite. The approach labels in Finding 5 come from a sample of attempts labelled by an agent and not yet checked by hand.

References

1. C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press and K. Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024.
2. M. A. Merrill, A. G. Shaw, et al. Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces ICLR 2026.
3. A. Ouyang, S. Guo, S. Arora, A. L. Zhang, W. Hu, C. Ré and A. Mirhoseini. KernelBench: Can LLMs Write Efficient GPU Kernels? ICML 2025.
4. A. Brohan et al. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control CoRL 2023.
5. M. J. Kim et al. OpenVLA: An Open-Source Vision-Language-Action Model CoRL 2024.
6. K. Black et al. π0: A Vision-Language-Action Flow Model for General Robot Control arXiv 2024.
7. J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence and A. Zeng. Code as Policies: Language Model Programs for Embodied Control ICRA 2023.

Data and citation

Tasks, results and agent runs are on this site (tasks, leaderboard, runs); videos and run records are in the media dataset.

@misc{embodiedfirstexam2026,
  title  = {Embodied First Exam},
  author = {Embodied First Exam Team},
  year   = {2026},
  url    = {https://www.embodied-first-exam.ai}
}