Harbor Hub

terminal-bench/terminal-bench

Terminal-Bench is a benchmark for measuring agents' abilities to complete tasks using a terminal.

New Job
harbor run -d terminal-bench/terminal-bench

Terminal-Bench

Terminal-Bench is wide and dynamic. Terminal-Bench measures general agent capabilities on a diverse set of difficult tasks.

Terminal-Bench will be continuously updated with new and improved tasks.

Run

Terminal-Bench includes GPU and multi-container tasks. We recommend running on Modal or Daytona.

First, install the Harbor CLI.

uv tool install "harbor[modal,daytona]"
# or pip install "harbor[modal,daytona]"

And then run:

harbor run -d terminal-bench/terminal-bench \
  --agent claude-code \
  --model anthropic/claude-fable-5 \
  --n-concurrent 100 \
  --env modal

To upload results to Harbor Hub (e.g. for leaderboard submission), include the --upload flag or run harbor upload "<job/path>".

Contribute

See CONTRIBUTING.md for how to propose, implement, and submit new tasks.

We also encourage the community to reach out when they find bugs in tasks. Building benchmarks for current model capabilities is hard — community feedback helps keep quality high.

Task
terminal-bench/ico-path-patch
terminal-bench/cad-model
terminal-bench/hof-topology-interpenetration
terminal-bench/memcached-backdoor
terminal-bench/mp-checkpoint-consolidation
terminal-bench/production-planning
terminal-bench/html-js-filter
terminal-bench/data-anonymization
terminal-bench/retro-console-soc
terminal-bench/interleaved-vigenere
terminal-bench/fin-saccr-rwa
terminal-bench/intrastat-meldung
terminal-bench/erp-procurement-planning
terminal-bench/rs-archive-clone
terminal-bench/wdm-design
terminal-bench/fix-uautomizer-soundness
terminal-bench/session-window-debug
terminal-bench/legacy-utility-triage
terminal-bench/lake-temp-glm
terminal-bench/coq-block-bound
terminal-bench/bun-sourcemap-leak
terminal-bench/payments-pipeline-fix
terminal-bench/foodstuff-beta-activity
terminal-bench/math-eval-grader
terminal-bench/jax-speedrun-gpu
terminal-bench/freecad-spring-clip
terminal-bench/mvcc-lsm-compaction
terminal-bench/batched-eval-parity
terminal-bench/vba-userform-port
terminal-bench/vf2-speedup-networkx
terminal-bench/distributed-dedup
terminal-bench/freecad-platform-drawing
terminal-bench/live-database-cutover
terminal-bench/cumulative-layout-shift
terminal-bench/nextjs-performance
terminal-bench/layout-config-recreation
terminal-bench/cli-2ph-simplex
terminal-bench/gsea-proteomics
terminal-bench/pretrain-shard-corruption
terminal-bench/layout-config-recreation2
terminal-bench/ks-solver-cpp
terminal-bench/gpt2-codegolf
terminal-bench/biped-contact-dynamics
terminal-bench/vllm-deepseek-streaming
terminal-bench/ctr-optimization
terminal-bench/cargo-flight-dispatch
terminal-bench/risk-scorer-replay
terminal-bench/formal-crypto
terminal-bench/vpp-loss-divergence
terminal-bench/glycan-ms2-elucidation
terminal-bench/photonic-waveguide-routing
terminal-bench/sound-change-cascade
terminal-bench/sglang-qwen-burst
terminal-bench/heat-pump-warranty
terminal-bench/fp8-rmsnorm-gemm
terminal-bench/telecom-entity-resolution
terminal-bench/lean-midpoint-proof
terminal-bench/freight-dispatch-shift
terminal-bench/takens-embedding-lean
terminal-bench/react-lead-form
terminal-bench/music-harmony
terminal-bench/exam-pdf-eval
terminal-bench/roy-polymorph-cn
terminal-bench/protein-autointerp-disulfide
terminal-bench/uefi-bootkit
terminal-bench/medical-claims-processing
terminal-bench/kv-live-surgery
terminal-bench/satb-audio-transcription
terminal-bench/atrx-vep-crispr
terminal-bench/ontology-kg-querying
terminal-bench/embedding-drift-monitor
terminal-bench/shadow-relay
terminal-bench/freecad-impeller
terminal-bench/wal-recovery-ordering

Displaying 74 of 74 tasks