Loading dataset details…
harbor run -d bencalvert04/programbenchharbor run -d bencalvert04/programbenchProgramBench cleanroom reconstruction tasks: reimplementing real-world CLI tools from popular open-source repos.
harbor run -d bencalvert04/programbenchProgramBench cleanroom reconstruction tasks: reimplementing real-world CLI tools from popular open-source repos.
harbor run -d bencalvert04/programbenchA Harbor port of ProgramBench: 200 cleanroom program-reconstruction tasks. In each task, an agent sees only a compiled reference binary and its documentation for a real-world open-source CLI tool, and must reimplement the program from scratch so its behavior matches the original. Scoring runs the agent's build against a hidden behavioral test suite; the reward is the pass rate over active tests.
bencalvert04/<repo-org>--<repo-name>.<commit> (e.g. bencalvert04/burntsushi--ripgrep.3b7fd44), one per source repository.testorg__ fixture task.network_mode = "allowlist" (LLM API hosts only)./workspace wiped, so the agent's own binary can't leak into scoring. Hidden tests are pulled from HuggingFace or a local blob cache.solution/solve.sh) restores the cleanroom binary after the hash-scrub, so Harbor's built-in oracle agent gives a sanity ceiling. Note the oracle mean across all 200 tasks is expected to be below 1.0 (upstream test flakiness / partial branch coverage on some tasks) — this is a known upstream property, not a bug in the port.Each task's reward.json reports three keys (all 0–1, averaged by Harbor's default Mean):
| Key | Meaning |
|---|---|
reward |
pass rate over active hidden tests |
resolved |
1.0 if pass rate is exactly 1.0, else 0.0 |
almost_resolved |
1.0 if pass rate is ≥ 0.95, else 0.0 |
harbor run -d bencalvert04/programbench -a claude-code -m anthropic/claude-opus-4-1Or against a single task:
harbor run -t bencalvert04/programbench/burntsushi--ripgrep.3b7fd44 -a claude-code -m anthropic/claude-opus-4-1Generated from a local ProgramBench checkout via the programbench-adapter CLI in this repo (adapters/programbench/). See that adapter's README for adapter internals, task-template details, and the full generation/verification harness.
A Harbor port of ProgramBench: 200 cleanroom program-reconstruction tasks. In each task, an agent sees only a compiled reference binary and its documentation for a real-world open-source CLI tool, and must reimplement the program from scratch so its behavior matches the original. Scoring runs the agent's build against a hidden behavioral test suite; the reward is the pass rate over active tests.
bencalvert04/<repo-org>--<repo-name>.<commit> (e.g. bencalvert04/burntsushi--ripgrep.3b7fd44), one per source repository.testorg__ fixture task.network_mode = "allowlist" (LLM API hosts only)./workspace wiped, so the agent's own binary can't leak into scoring. Hidden tests are pulled from HuggingFace or a local blob cache.solution/solve.sh) restores the cleanroom binary after the hash-scrub, so Harbor's built-in oracle agent gives a sanity ceiling. Note the oracle mean across all 200 tasks is expected to be below 1.0 (upstream test flakiness / partial branch coverage on some tasks) — this is a known upstream property, not a bug in the port.Each task's reward.json reports three keys (all 0–1, averaged by Harbor's default Mean):
| Key | Meaning |
|---|---|
reward |
pass rate over active hidden tests |
resolved |
1.0 if pass rate is exactly 1.0, else 0.0 |
almost_resolved |
1.0 if pass rate is ≥ 0.95, else 0.0 |
harbor run -d bencalvert04/programbench -a claude-code -m anthropic/claude-opus-4-1Or against a single task:
harbor run -t bencalvert04/programbench/burntsushi--ripgrep.3b7fd44 -a claude-code -m anthropic/claude-opus-4-1Generated from a local ProgramBench checkout via the programbench-adapter CLI in this repo (adapters/programbench/). See that adapter's README for adapter internals, task-template details, and the full generation/verification harness.