AutoResearchExam

AutoResearchExam measures an agent's ability to improve and generalize on open ended machine learning research tasks. The dataset contains 29 tasks across machine learning research.

View tasks by area →

Model training

Representation learning

• Compact encoding for a column with many categories
• Compact item embeddings for recommendations
• Short binary codes for image search

Reinforcement learning

• Advantage estimation for policy training
• Reliable control policy trainer

Predictive modeling

• Predict valid actions for a game policy
• Rank employee access requests from ID codes

Model merging and distillation

• Eight domain vision model merging

Algorithms and optimization

Statistical methods

• Estimate uncertainty in a streaming optimizer
• Estimate individual treatment effects
• Recover river flow links

Constrained optimization

• Dual market budgeted classification
• MILP branching policy
• Online chance constrained policy

Data engineering and curation

Imputation

• Budgeted imputation
• Sparse panel imputation

Data selection

• Unlabeled pool pruning
• Low discrepancy subset selection
• Group robust coreset selection
• Pretraining token budget selection

Systems and efficiency

Inference

• CPU language model decoding
• CPU decoder graph execution
• Video diffusion transformer caching

Compression

• SVDQuant PixArt Sigma quantization

Evaluation, calibration, and robustness

Metric estimation

• Shortest valid confidence interval
• Label efficient risk estimation

Robustness

• Adversarially robust CIFAR 10 training

AI safety and alignment

• Candidate token loss ranker

Interpretability

• Sparse dictionary coding

Read the benchmark and results

View the tasks on GitHub

Run with the harness

Download the tasks and harness into separate folders:

mkdir autoresearch-exam-run && cd autoresearch-exam-run
git clone https://github.com/bespokelabsai/AutoResearchExam-Terminus.git harness
cd harness
uv sync --python 3.12 --extra modal
uv run harbor dataset download bespokelabs/autoresearch-exam@latest \
  --output-dir ../harbor-tasks \
  --export

Run one task on Modal:

uv run harbor run \
  --path ../harbor-tasks/autoresearch-exam/cpu-decoder-graph-executor \
  --agent autoresearchexam-terminus \
  --model openai/gpt-5.6-sol \
  --env modal \
  --plugin autoresearch-exam \
  --pk max_iterations=1 \
  --pk max_duration_seconds=3600

Use --path ../harbor-tasks/autoresearch-exam to run all tasks. Use --env docker to run with local Docker.

AutoResearchExam-Terminus provides the harness and the code used to compute AUARC.