Agents' Last Exam
for Robotics
The robotics domain of Agents' Last Exam. Autonomous agents are dropped into sandboxed robotics problems and scored on the artifact they leave behind — by a hidden grader, on hidden seeds.
Four commitments that make results meaningful
The Agentic Robotics Benchmark is built so that a task's score means what it says.
One kind of task: agentic robotics
Each task drops an autonomous agent into a sandboxed robotics problem — develop a controller, plan a manipulation sequence, design a mechanism, write code-as-policy, tune a planner — and scores whatever artifact it leaves behind: a controller process, a plan, a design file, generated code. There is no answer key to retrieve.
Grader-computed metrics
The authoritative metric is computed by the task's verify stage on hidden seeds, in a stage the agent never sees. Nothing the agent prints is a scoring input.
Hidden by isolation, not by secrecy
Everything the grader needs — the anchor included — travels inside the task folder and is kept from the agent by stage isolation: verify/ is simply absent while the agent works. Anchors are published here.
Open, versioned tasks
Every task is a folder in the public benchmark repository. The engine's own loader identifies each version by hashes of its spec and its files, and this site shows exactly what the benchmark exports at a recorded commit.
Organized by platform
Every task belongs to one platform of the benchmark's vocabulary and scores one headline metric against the anchor its reference implementation measured.
Robotic arm
0No tasks yet
Dexterous hand
0No tasks yet
Quadruped
0No tasks yet
Humanoid
0No tasks yet
UAV
0No tasks yet
Autonomous driving
0No tasks yet
Underwater robot (UUV/AUV)
0No tasks yet
Space robot
1Soft robot
0No tasks yet
Surgical robot
0No tasks yet
Swarm
0No tasks yet
Subsystems
0No tasks yet
Bring a robotics problem into the benchmark.
The benchmark grows through contributed tasks — each one a self-contained folder (instruction, image, reference oracle, hidden verifier) that scores exactly zero untouched and exactly one with its oracle. A task is contributed as a pull request to the benchmark repository; the submission guidelines explain how to take part.