Two ways to submit, one result: a pull request into the benchmark repository that adds your task folder at tasks/<direction>/<slug>/. Steps 1 to 3 are the same either way. In step 5 you open that pull request yourself, or upload the zipped folder here and a bot opens it for you. The maintainers review both the same way.

1. 01

   Get the engine

   Tasks are built and checked with the ALE engine. It is a public Git repository, not a package; you need Python 3.12, uv and Docker.

   git clone https://github.com/AgentsLastExam/ale.git && cd ale
   uv sync                                    # Python 3.12+ and uv; ale is not on PyPI
   uv run ale --help

   Build the base image once. Every task image starts from it.

   images/build.sh container-ubuntu22          # from the engine checkout; slow the first time

2. 02

   Download the template and fill it in

   Download agentic-robotics-task-template.ziptemplates/task @ b04359f1b9… · built Sep 8, 2026 · sha256 e1ccf56197ba…

   Unzip it. Rename the task folder to your slug (lowercase letters, digits, _ and -) and set name in task.yaml to the same string. Fill in task.yaml (platform, direction, a hardware-invariant metric), instruction.md, image/Dockerfile, setup/, oracle/ and verify/. The agent leaves any artifact in /home/user/submission/; verify/ scores it, oracle/ is the reference that scores full. The format and the bar are the engine's: Writing a standard task and Task quality standard.

   unzip agentic-robotics-task-template.zip
   mv agentic-robotics-task-template my_task  # your slug; task.yaml `name` must equal it
   $EDITOR my_task/task.yaml                  # name, metadata.platform / direction / metric

3. 03

   Lint and validate

   ale lint checks the files. ale validate builds the image, runs an untouched sandbox and the oracle, and requires a zero for the first and exactly one for the second. The first validate fails with oracle_not_full: the template ships a null anchor. Read the measured metric from the oracle's result.json, write it into verify/anchor.json as value with its source (your run, never a paper's figure) and full_at, then validate again until it passes.

   # from the engine checkout; <task_dir> = the absolute path of my_task, <runs> = a scratch directory
   uv run ale lint <task_dir>
   uv run ale validate <task_dir> --runs-dir <runs>   # fails with oracle_not_full while the anchor is null
   cat <runs>/validate-*/<slug>-oracle-*/result.json      # the measured metric, under "metrics"
   $EDITOR <task_dir>/verify/anchor.json      # value = that number; source = your run; full_at
   uv run ale validate <task_dir> --runs-dir <runs>   # repeat until it passes

4. 04

   Zip the task folder (for the upload)

   Only needed if you upload here rather than opening the pull request yourself. Zip the task folder alone, not the download root. Limits: 50 MB total, 256 files, 2 MB per file, no symlinks; file names use letters, digits, ., _, - and /. Large assets are fetched by the image, not zipped.

   cd <the directory that contains my_task>   # not inside my_task
   zip -r my_task.zip my_task
   unzip -l my_task.zip                        # every entry starts with my_task/

5. 05

   Open the pull request, or upload here

   First way: open it yourself. Fork the benchmark repository, copy your folder to tasks/<direction>/<slug>/ (<direction> is metadata.direction from your task.yaml), push a branch and open the pull request. CI runs ale lint, the domain lint and the grader-kit comparison on it; a maintainer re-runs ale validate and merges.

   git clone https://github.com/<you>/agentic-robotics-benchmark.git   # your fork of the benchmark repository
   cd agentic-robotics-benchmark && git switch -c add-my_task
   cp -r <the directory that contains my_task>/my_task tasks/<direction>/my_task   # <direction> = metadata.direction in task.yaml
   git add tasks/<direction>/my_task && git commit -m "Add task <direction>/my_task"
   git push -u origin add-my_task              # then open the pull request on GitHub

   Second way: upload here. The server runs ale lint on your archive, checks the slug, the metadata, the anchor and the file names, then opens the same pull request for you and shows you the link. The slug must be new. A failed check reports every problem and stores nothing. Check only (nothing is stored) runs the same checks without uploading. Follow the review on the pull request or on your profile; a revision is a new zip with the same slug.

   Checking session…

   Title*

   3–256 characters; shown to maintainers and on your profile. Not needed for a check.

   Task archive (.zip)*

   50 MB max, 256 files, 2 MB per file, no symlinks. Checked here before the upload starts; the server re-checks everything.

   Upload for reviewCheck only (nothing is stored)Sign in to enable the upload and the check.