TB2.1 Code & Debug Track

Coding, software engineering, debugging, build repair, language/runtime work, and game-like reasoning tasks from Terminal-Bench 2.1.

This public Harbor dataset is curated by NovitaAI for a Novita Sandbox hackathon track. It is a category-based subset of Terminal-Bench 2.1. Agents and models are not fixed; the only required runtime environment for the hackathon is Novita Sandbox.

Dataset

• Harbor dataset: NovitaAI/tb21-code-debug
• Track size: 32 tasks
• Source benchmark: Terminal-Bench 2.1
• Included source categories: software-engineering, debugging, games
• Required hackathon sandbox: -e novita

Quick Start

Run the full track once:

harbor run \
  -d NovitaAI/tb21-code-debug \
  -a <agent> \
  -m <model> \
  -e novita \
  -k 1 \
  -n 1 \
  -y

Run a small smoke test from the track:

harbor run \
  -d NovitaAI/tb21-code-debug \
  -a <agent> \
  -m <model> \
  -e novita \
  -l 1 \
  -k 1 \
  -n 1 \
  -y

Upload a public result for the hackathon leaderboard:

harbor upload jobs/<job_name> --public

Submit the resulting Harbor Hub job link to the hackathon leaderboard form.

Valid Submission Rules

A valid track submission should satisfy:

• The Harbor job uses this dataset: NovitaAI/tb21-code-debug.
• The job config has environment.type = "novita".
• The job does not use extra hints or task-specific extra instructions.
• The submitted Harbor Hub job is public.
• Agent and model are free choice unless a specific event round says otherwise.

Suggested ranking fields:

• Primary: mean reward
• Tie-breaker 1: fewer exceptions/errors
• Tie-breaker 2: lower average duration
• Tie-breaker 3: lower output tokens or total tokens, if the event wants an efficiency prize

Tasks

• terminal-bench/build-cython-ext
• terminal-bench/build-pmars
• terminal-bench/build-pov-ray
• terminal-bench/cancel-async-tasks
• terminal-bench/chess-best-move
• terminal-bench/circuit-fibsqrt
• terminal-bench/cobol-modernization
• terminal-bench/code-from-image
• terminal-bench/custom-memory-heap-crash
• terminal-bench/fix-git
• terminal-bench/fix-ocaml-gc
• terminal-bench/git-leak-recovery
• terminal-bench/gpt2-codegolf
• terminal-bench/headless-terminal
• terminal-bench/kv-store-grpc
• terminal-bench/make-doom-for-mips
• terminal-bench/make-mips-interpreter
• terminal-bench/merge-diff-arc-agi-task
• terminal-bench/overfull-hbox
• terminal-bench/path-tracing
• terminal-bench/path-tracing-reverse
• terminal-bench/polyglot-c-py
• terminal-bench/polyglot-rust-c
• terminal-bench/prove-plus-comm
• terminal-bench/pypi-server
• terminal-bench/regex-chess
• terminal-bench/schemelike-metacircular-eval
• terminal-bench/sqlite-db-truncate
• terminal-bench/torch-pipeline-parallelism
• terminal-bench/torch-tensor-parallelism
• terminal-bench/winning-avg-corewars
• terminal-bench/write-compressor