MGD a suite of the exam · One arm
ManiGuard
Manipulation safety. A task counts only if no safety rule was broken on the way.
A manipulation safety benchmark from the NU IDEAS Lab: tabletop tasks for a Franka Panda with long fin-ray fingers, each with a safety specification in linear temporal logic over physics predicates (dropped, upright, on top of, closed, touching), checked after every control step. A task counts only if the goal holds and no rule was ever broken.
| Origin | adapted from the upstream benchmark |
|---|---|
| Simulator | BEHAVIOR-1K v3.7.2 (OmniGibson 3.7.2 on Isaac Sim 4.5) |
| Robot | Franka Panda |
| Tasks | 30 tasks · 60 tests |
| Agent time | 60 min |
On a real robot
7 tasks
1
Claude Opus 5.5Claude Code · high
100%7/7
2
Claude Fable 5.2Claude Code · high
86%6/7
3
GPT-6.1 SolCodex · high
71%5/7
4
Gemini 4 ArgonAntigravity · high
57%4/7
4
GPT-6.2 AstraCodex · high
57%4/7
6
GPT-6 AstraCodex · high
43%3/7
6
Kimi 3.1 ProClaude Code · high
43%3/7
8
DeepSeek V4.1 FlashDeepSeek Harness · high
0%0/7
8
Gemini 3.8 FlashAntigravity · high
0%0/7
8
GLM 5.3 FlashClaude Code · high
0%0/7
02
Tasks
30 tasks · 60 tests