ISG  a suite of the exam · Mobile robot

iSpatialGymEFE Original

Tests spatial reasoning from images. The goal is given only as a picture: a mobile two-armed robot rearranges the room, or moves itself, until its camera view matches it.

Spatial reasoning for a mobile two-armed robot, a Galaxea R1 Pro, in BEHAVIOR-1K's houses. Each task gives only a goal image, the head camera's view once the task is done, and no sentence names the objects: line an object up with a table edge, set a tilted one upright, place it in, on, under, next to or touching another, open an appliance to the pictured angle, arrange blocks into a pattern, turn a held object to the pictured orientation, or drive and move until its own camera sees the goal view. Success is a geometric check: every measured target within tolerance, with the objects and the robot at rest over the last moments of the episode. No reference solution ships with these tasks.

EFE Original
Originbuilt by the exam’s team, released with the exam
SimulatorOmniGibson (BEHAVIOR-1K fork) on Isaac Sim 5.1
RobotGalaxea R1 Pro
Tasks12 tasks · 24 tests
Agent time60 min
01

Results

8 tasks
1 GPT-6.2 AstraCodex · high 88%7/8
2 Claude Fable 5.2Claude Code · high 75%6/8
3 GPT-6 AstraCodex · high 50%4/8
3 GPT-6.1 SolCodex · high 50%4/8
5 Claude Opus 5.5Claude Code · high 38%3/8
6 Gemini 4 ArgonAntigravity · high 25%2/8
7 Gemini 3.8 FlashAntigravity · high 13%1/8
7 Kimi 3.1 ProClaude Code · high 13%1/8
9 DeepSeek V4.1 FlashDeepSeek Harness · high 0%0/8
9 GLM 5.3 FlashClaude Code · high 0%0/8
02

Tasks

12 tasks · 24 tests