Loading dataset details…
harbor run -d bespokelabs/autoresearch-examharbor run -d bespokelabs/autoresearch-examAutoResearchExam measures an agent's ability to improve and generalize on open ended machine learning research tasks. The dataset contains 29 tasks across machine learning research.
harbor run -d bespokelabs/autoresearch-examAutoResearchExam measures an agent's ability to improve and generalize on open ended machine learning research tasks. The dataset contains 29 tasks across machine learning research.
harbor run -d bespokelabs/autoresearch-examAutoResearchExam measures an agent's ability to improve and generalize on open ended machine learning research tasks. The dataset contains 29 tasks across machine learning research.
Read the benchmark and results
Download the tasks and harness into separate folders:
mkdir autoresearch-exam-run && cd autoresearch-exam-run
git clone https://github.com/bespokelabsai/AutoResearchExam-Terminus.git harness
cd harness
uv sync --python 3.12 --extra modal
uv run harbor dataset download bespokelabs/autoresearch-exam@latest \
--output-dir ../harbor-tasks \
--exportRun one task on Modal:
uv run harbor run \
--path ../harbor-tasks/autoresearch-exam/cpu-decoder-graph-executor \
--agent autoresearchexam-terminus \
--model openai/gpt-5.6-sol \
--env modal \
--plugin autoresearch-exam \
--pk max_iterations=1 \
--pk max_duration_seconds=3600Use --path ../harbor-tasks/autoresearch-exam to run all tasks. Use --env docker to run with local Docker.
AutoResearchExam-Terminus provides the harness and the code used to compute AUARC.
AutoResearchExam measures an agent's ability to improve and generalize on open ended machine learning research tasks. The dataset contains 29 tasks across machine learning research.
Read the benchmark and results
Download the tasks and harness into separate folders:
mkdir autoresearch-exam-run && cd autoresearch-exam-run
git clone https://github.com/bespokelabsai/AutoResearchExam-Terminus.git harness
cd harness
uv sync --python 3.12 --extra modal
uv run harbor dataset download bespokelabs/autoresearch-exam@latest \
--output-dir ../harbor-tasks \
--exportRun one task on Modal:
uv run harbor run \
--path ../harbor-tasks/autoresearch-exam/cpu-decoder-graph-executor \
--agent autoresearchexam-terminus \
--model openai/gpt-5.6-sol \
--env modal \
--plugin autoresearch-exam \
--pk max_iterations=1 \
--pk max_duration_seconds=3600Use --path ../harbor-tasks/autoresearch-exam to run all tasks. Use --env docker to run with local Docker.
AutoResearchExam-Terminus provides the harness and the code used to compute AUARC.