Contribute · Embodied First Exam

Five ways in

From a whole suite to a single bug report.

Propose a suite

Bring a benchmark under the protocol, in both modes.

Fill in the proposal ↓ Add tasks

A new family or more scenes for a suite already in.

Kinds of pull requests ↗ Review a family

Check a family that has no full reference solution.

The review checklist ↗ Break a task

Pass a verifier without doing the task, then help fix it.

Task fixes ↗ Run batches

Run assigned batches on your GPUs and send the logs back.

Sending back runs ↗

What a suite needs

Six things every proposal shows.

VerifiableA program decides success, with no learned judge.
SolvableFor every family, one answer the verifier marks 1.
DifficultHard for today’s code agents, for a robotic reason.
ReproducibleA frozen scene that replays bit for bit.
IsolatableMarked in its own sandbox from the answer alone.
ScrubbableReference solutions kept out of the agent’s reach.

From proposal to leaderboard

Six steps. The protocol ↗ has every detail.

ProposeOpen an issue, or fill in the form below.proposal-approved
BuildTwo task folders per frozen scene, from a generator.-privileged-standard
Self-checkRun the checks the reviewers will run.run_all.shoracle = 1nop = 02 replays
ReviewA maintainer, a cheat trial and a model trial.cheat trialhuman review
MergeBase images published, the suite added to the site.eai-<name>
On the boardIts questions join the exam’s evaluations.both modesevery log

Propose a suite

Fill in the fields of a proposal and copy it as Markdown. Nothing is sent from this page.

Where a new suite helps most

Labels and robot bodies with room for more questions.

Credit

How contributions could count toward authorship of the exam’s paper.

House rules

From the contributing guide and the protocol.

A published task never changes. Fixing a scene makes a new task.
Everything is pinned, and marking needs no network.
The agent is root in its container, so marking runs in another.
Reference solutions live only under solution/.
No credentials, personal paths, large binaries or results in the repository.
One benchmark per pull request; shared files and protocol changes go apart.
An image that cannot be shared stays build-only, with build steps.
Official runs use Harbor’s native agents, with web search off.

Questions

How do I send a proposal?

Fill in the form above and copy it as Markdown, then send it to a maintainer you know. The exam’s repository is not public yet; access for the pull request is arranged once a proposal is approved.

My benchmark has no scripted expert. Can it still come in?

Yes. A family without a full reference solution needs a positive example (a successful model trial, an upstream policy rollout or a hand-built trajectory) and a human review. Its results are reported apart from the tasks with a verified solution until it has one.

Does my suite need both modes?

Each frozen scene is offered as a privileged and a standard task. Standard mode needs a service that runs the simulator behind the standard protocol (eai-standard/2.x); the services of the suites already in are the templates.

Which simulators are accepted?

Any that runs headless in a pinned Docker image and replays a frozen scene deterministically on the reference hardware, an RTX 4090. The exam already runs MuJoCo, Isaac Sim, SAPIEN and Genesis, among others.

What if the simulator or its assets cannot be redistributed?

The image is marked build-only and comes with build instructions (licences, data). It is never published, though the maintainers may keep a private package for the team.

Can I get my own agent on the leaderboard?

Official results come from Harbor’s native agents (Codex, Claude Code and others) under the protocol’s budgets, run with our runner. Helpers with GPU hosts run assigned batches and send the job directories back; ask the maintainers for an assignment.

How is a suite credited?

Every task names its author, and every adapted suite credits its upstream benchmark on its suite page and in its documentation.