Loading dataset details…
harbor run -d bespokelabs/autoresearch-examharbor run -d bespokelabs/autoresearch-examAutoResearchExam measures an agent's ability to improve and generalize on open ended machine learning research tasks. The dataset contains 29 tasks across machine learning research.
harbor run -d bespokelabs/autoresearch-examAutoResearchExam measures an agent's ability to improve and generalize on open ended machine learning research tasks. The dataset contains 29 tasks across machine learning research.
harbor run -d bespokelabs/autoresearch-examThis repository contains the 29 tasks from AutoResearchExam. Version 1.1 runs natively on Harbor with any agent harness. Leaderboard scores come from the official harness on version 1.0. See Run for both.
harbor run
on Harbor main, without depending on any
custom harness. Same tasks, data and graders as v1.0. Leaderboard scores still come from
v1.0 with the official harness.| Research area | Tasks |
|---|---|
| Model training | 8 |
| Algorithms and optimization | 6 |
| Data engineering and curation | 6 |
| Systems and efficiency | 4 |
| Evaluation, calibration, and robustness | 3 |
| AI safety and alignment | 1 |
| Interpretability | 1 |
| Total | 29 |
Each run has a 24 hour research budget, and the agent should keep working until it ends.
The leaderboard uses the v1.0 tasks with the official AutoResearchExam-Terminus harness. To reproduce leaderboard scores, follow the instructions in that repository.
The official harness keeps the agent working and submitting until the budget ends. With your own harness, that is up to the agent. An agent can stop early, but then it scores lower, because it did not follow the instruction to keep going.
Install Harbor, start Docker, and set up your model provider's credentials. Then run any agent harness on a task:
git clone https://github.com/bespokelabsai/AutoResearchExam.git
harbor run \
--path AutoResearchExam/cpu-decoder-graph-executor \
--agent <agent> --model <model> --env docker \
--ak prompt_template_path=AutoResearchExam/are_protocol.md.j2Use --path AutoResearchExam to run all tasks. are_protocol.md.j2 adds the research protocol
after each task instruction: submit regularly, keep improving the solution, and keep going
until the run ends.
During the run, the agent can reach api.openai.com and api.anthropic.com. If your harness
uses another host, add --allow-agent-host <host>.
GPU tasks need an NVIDIA GPU and the NVIDIA Container Toolkit. Grading shares GPU 0 with the
agent by default. On a machine with a spare GPU, set ARE_GRADER_GPU=1 to give grading its
own GPU.
Each task runs the agent next to two grader containers.
submit tool of the are MCP server, or over HTTP at
http://grader:8000/submit. The first grader scores the submission on the public
validation split. The agent gets the score and grader output from the result tool.In v1.0, the official harness keeps the agent submitting for the full 24 hours. In v1.1, that is your agent's responsibility.
In v1.0, grading paused the agent. Now the agent keeps working while a submission is graded, so it has more working time. Because only one submission is graded at a time, an agent cannot flood the grader.
Grading runs on the same machine as the agent and can compete with it for compute. A grader uses as many cores as the task gives the agent, and the two graders never run at the same time, so plan for about twice the cores each task requests.
Each trial writes verifier/reward.json. Its reward is the private test score of the
submission with the best validation score. Its auarc uses each task's own score scale. To
put scores on the leaderboard scale, follow the
scoring guide
to normalize scores and compute AUARC. Every submission's scores and times are in the
trial's artifacts/var/lib/are/ledger.jsonl.
If you use AutoResearchExam in your research, please cite:
@misc{ramesh2026autoresearchexam,
author = {Ramesh, Anirudha and
Devic, Siddartha and
Garg, Shivank and
Parulekar, Advait and
Mahnot, Drish and
Pimpalgaonkar, Shreyas and
Suresh, Vishnu and
Dimakis, Alex and
Sathiamoorthy, Maheswaran},
title = {{AutoResearchExam}},
year = {2026},
month = sep,
url = {https://benchmarks.bespokelabs.ai/autoresearchexam/},
note = {Published September 9, 2026}
}
This repository contains the 29 tasks from AutoResearchExam. Version 1.1 runs natively on Harbor with any agent harness. Leaderboard scores come from the official harness on version 1.0. See Run for both.
harbor run
on Harbor main, without depending on any
custom harness. Same tasks, data and graders as v1.0. Leaderboard scores still come from
v1.0 with the official harness.| Research area | Tasks |
|---|---|
| Model training | 8 |
| Algorithms and optimization | 6 |
| Data engineering and curation | 6 |
| Systems and efficiency | 4 |
| Evaluation, calibration, and robustness | 3 |
| AI safety and alignment | 1 |
| Interpretability | 1 |
| Total | 29 |
Each run has a 24 hour research budget, and the agent should keep working until it ends.
The leaderboard uses the v1.0 tasks with the official AutoResearchExam-Terminus harness. To reproduce leaderboard scores, follow the instructions in that repository.
The official harness keeps the agent working and submitting until the budget ends. With your own harness, that is up to the agent. An agent can stop early, but then it scores lower, because it did not follow the instruction to keep going.
Install Harbor, start Docker, and set up your model provider's credentials. Then run any agent harness on a task:
git clone https://github.com/bespokelabsai/AutoResearchExam.git
harbor run \
--path AutoResearchExam/cpu-decoder-graph-executor \
--agent <agent> --model <model> --env docker \
--ak prompt_template_path=AutoResearchExam/are_protocol.md.j2Use --path AutoResearchExam to run all tasks. are_protocol.md.j2 adds the research protocol
after each task instruction: submit regularly, keep improving the solution, and keep going
until the run ends.
During the run, the agent can reach api.openai.com and api.anthropic.com. If your harness
uses another host, add --allow-agent-host <host>.
GPU tasks need an NVIDIA GPU and the NVIDIA Container Toolkit. Grading shares GPU 0 with the
agent by default. On a machine with a spare GPU, set ARE_GRADER_GPU=1 to give grading its
own GPU.
Each task runs the agent next to two grader containers.
submit tool of the are MCP server, or over HTTP at
http://grader:8000/submit. The first grader scores the submission on the public
validation split. The agent gets the score and grader output from the result tool.In v1.0, the official harness keeps the agent submitting for the full 24 hours. In v1.1, that is your agent's responsibility.
In v1.0, grading paused the agent. Now the agent keeps working while a submission is graded, so it has more working time. Because only one submission is graded at a time, an agent cannot flood the grader.
Grading runs on the same machine as the agent and can compete with it for compute. A grader uses as many cores as the task gives the agent, and the two graders never run at the same time, so plan for about twice the cores each task requests.
Each trial writes verifier/reward.json. Its reward is the private test score of the
submission with the best validation score. Its auarc uses each task's own score scale. To
put scores on the leaderboard scale, follow the
scoring guide
to normalize scores and compute AUARC. Every submission's scores and times are in the
trial's artifacts/var/lib/are/ledger.jsonl.
If you use AutoResearchExam in your research, please cite:
@misc{ramesh2026autoresearchexam,
author = {Ramesh, Anirudha and
Devic, Siddartha and
Garg, Shivank and
Parulekar, Advait and
Mahnot, Drish and
Pimpalgaonkar, Shreyas and
Suresh, Vishnu and
Dimakis, Alex and
Sathiamoorthy, Maheswaran},
title = {{AutoResearchExam}},
year = {2026},
month = sep,
url = {https://benchmarks.bespokelabs.ai/autoresearchexam/},
note = {Published September 9, 2026}
}