bencalvert04/programbench

A Harbor port of ProgramBench: 200 cleanroom program-reconstruction tasks. In each task, an agent sees only a compiled reference binary and its documentation for a real-world open-source CLI tool, and must reimplement the program from scratch so its behavior matches the original. Scoring runs the agent's build against a hidden behavioral test suite; the reward is the pass rate over active tests.

What's in the dataset

• 200 tasks, each named bencalvert04/<repo-org>--<repo-name>.<commit> (e.g. bencalvert04/burntsushi--ripgrep.3b7fd44), one per source repository.
• Source languages: Rust (107), Go (46), C (33), C++ (12), Java (1), Haskell (1).
• Difficulty (as labeled upstream): 120 medium, 35 unknown, 27 easy, 18 hard.
• Excludes the upstream testorg__ fixture task.

How each task works

1. Agent phase — runs in a cleanroom container containing only the reference binary and docs, with network_mode = "allowlist" (LLM API hosts only).
2. Verifier phase — runs in a separate sandbox built from the same cleanroom image with /workspace wiped, so the agent's own binary can't leak into scoring. Hidden tests are pulled from HuggingFace or a local blob cache.
3. Oracle — a reference solution (solution/solve.sh) restores the cleanroom binary after the hash-scrub, so Harbor's built-in oracle agent gives a sanity ceiling. Note the oracle mean across all 200 tasks is expected to be below 1.0 (upstream test flakiness / partial branch coverage on some tasks) — this is a known upstream property, not a bug in the port.

Each task's reward.json reports three keys (all 0–1, averaged by Harbor's default Mean):

Key	Meaning
reward	pass rate over active hidden tests
resolved	1.0 if pass rate is exactly 1.0, else 0.0
almost_resolved	1.0 if pass rate is ≥ 0.95, else 0.0

Running it

harbor run -d bencalvert04/programbench -a claude-code -m anthropic/claude-opus-4-1

Or against a single task:

harbor run -t bencalvert04/programbench/burntsushi--ripgrep.3b7fd44 -a claude-code -m anthropic/claude-opus-4-1

Source

Generated from a local ProgramBench checkout via the programbench-adapter CLI in this repo (adapters/programbench/). See that adapter's README for adapter internals, task-template details, and the full generation/verification harness.