MGD  a suite of the exam · One arm

ManiGuard

Manipulation safety. A task counts only if no safety rule was broken on the way.

A manipulation safety benchmark from the NU IDEAS Lab: tabletop tasks for a Franka Panda with long fin-ray fingers, each with a safety specification in linear temporal logic over physics predicates (dropped, upright, on top of, closed, touching), checked after every control step. A task counts only if the goal holds and no rule was ever broken.

Originadapted from the upstream benchmark
SimulatorBEHAVIOR-1K v3.7.2 (OmniGibson 3.7.2 on Isaac Sim 4.5)
RobotFranka Panda
Tasks30 tasks · 60 tests
Agent time60 min
01

Results

7 tasks
1 Claude Opus 5.5Claude Code · high 100%7/7
2 Claude Fable 5.2Claude Code · high 86%6/7
3 GPT-6.1 SolCodex · high 71%5/7
4 Gemini 4 ArgonAntigravity · high 57%4/7
4 GPT-6.2 AstraCodex · high 57%4/7
6 GPT-6 AstraCodex · high 43%3/7
6 Kimi 3.1 ProClaude Code · high 43%3/7
8 DeepSeek V4.1 FlashDeepSeek Harness · high 0%0/7
8 Gemini 3.8 FlashAntigravity · high 0%0/7
8 GLM 5.3 FlashClaude Code · high 0%0/7
02

Tasks

30 tasks · 60 tests