Contribute
Call for contributors
The exam is an open agentic robotics benchmark, and it grows with its community. You can evaluate a model, integrate an agent harness, integrate tasks, or bring anything else that would make the exam better. All are very welcome, from labs, companies and individual researchers alike, and we would love to hear from you: here is how to contribute.
- 35
- suites in the exam
- 47
- robots
- 1,156
- tasks2,294 tests
How to contribute
Evaluate a model
Have a new model? We run it on the exam with an existing harness, under the same budgets as every other agent, and its results join the leaderboard.
Get in touch by e-mail →Integrate a harness
Bring your agent harness to the exam, so that any model can be tested in it and compared with Codex, Claude Code and the other harnesses on the same tests.
Get in touch by e-mail →Integrate tasks
Add a single task or a whole benchmark suite: frozen scenes, a program that decides success and, where there is one, a reference solution. What a suite needs.
Get in touch by e-mail →Something else
Real-robot evaluation, analysis of the agents’ runs, tools, or an idea we have not thought of: tell us what you have in mind and we will find a way to make it part of the exam.
Get in touch by e-mail →All go to embodiedfirstexam@gmail.com. For questions, join the exam’s Discord or its .
What a suite needs
- VerifiableA program decides success, with no learned judge.
- SolvableFor every family, one answer the verifier marks 1.
- DifficultHard for today’s code agents, for a robotic reason.
- ReproducibleA frozen scene that replays bit for bit.
- IsolatableMarked in its own sandbox from the answer alone.
- ScrubbableReference solutions kept out of the agent’s reach.
From proposal to leaderboard
- ProposeE-mail a short proposal.proposal-approved
- BuildTwo tests per task, each a task folder, from a generator.-privileged-standard
- Self-checkRun the checks the reviewers will run.run_all.shoracle = 1nop = 02 replays
- ReviewA maintainer, a cheat trial and a model trial.cheat trialhuman review
- MergeBase images published, the suite added to the site.eai-<name>
- On the boardIts tests join the exam’s evaluations.both modesevery log
Where a new suite helps most
Credit
Have an idea? Let’s talk.
A model, a harness, a task or anything else, write to us and we will work it out together.