Loading dataset details…
harbor run -d huu256/gauntlet-plannerharbor run -d huu256/gauntlet-plannerGauntlet: five hard fundraising-desk games (6x6 to 14x14), a stochastic job-shop scheduling problem played through a chat with a hidden game server. Pass/fail against a fixed bar. Same games with the optional @planner advice assistant enabled.
harbor run -d huu256/gauntlet-plannerGauntlet: five hard fundraising-desk games (6x6 to 14x14), a stochastic job-shop scheduling problem played through a chat with a hidden game server. Pass/fail against a fixed bar. Same games with the optional @planner advice assistant enabled.
harbor run -d huu256/gauntlet-plannerA planning benchmark played entirely through a chat. The agent runs an accelerator's fundraising desk: N founders each have to get past M gatekeepers (lawyer, lead VC, angels, accountant, ...) in their own order, and a gatekeeper reviews one founder at a time and never stops a review once started. Each founder has a round value, a weekly burn and a cash runway; the goal is the highest total NPV at a 30% annual hurdle rate. Underneath it is a stochastic job-shop scheduling problem with hidden, noisy information.
The agent learns everything from the chat (@chat help, @chat briefing, @chat board). It can ask a limited number
of questions per week (cash, burn, round size, review estimates), set gatekeeper queues (strict or loose), drop
founders, and step time (@chat step N). The game is turn-based: nothing moves until the agent steps. The agent has
one hour of real time from its first message; after that, open items play out on the orders it left.
| Task | Size | Questions/week |
|---|---|---|
| gauntlet-6x6-s6003 | 6 founders x 6 gatekeepers | 4 |
| gauntlet-8x8-s8013 | 8 x 8 | 7 |
| gauntlet-10x10-s10007 | 10 x 10 | 10 |
| gauntlet-14x14-s14001 | 14 x 14 | 12 |
| gauntlet-14x14-s14004 | 14 x 14 | 12 |
The gauntlet-planner dataset has the same games with @planner, an advice-only planning assistant that plans from
the agent's own stated beliefs. It is mentioned only in @chat help, never in the instruction.
Pass/fail. Each game's bar is the best of several fixed briefing-only players (first-come-first-served, one
biggest-round-first priority list strict or loose, and that list after dropping the best number of smallest rounds at
week 0) plus 5% of the remaining gap to the proven optimum. The bar is not shown during play; the chat's final message
reports the outcome, the bar and PASS/FAIL. reward.json also carries score_vs_naive (0 = best fixed player,
1 = optimum) as a continuous diagnostic.
Two containers. main holds only the chat client and an empty /workspace (Python 3.12 with numpy, scipy and
OR-Tools). The game server runs in a chatd sidecar that the agent reaches only over the chat port; the instance,
noise seed and hidden numbers never enter main. The verifier's control token is uploaded only with the tests, after
the agent finishes, and the server refuses to return results before the game ends.
Each task's solution/solve.sh replays, message for message, a recorded agent run that passed the game. The game is
deterministic given its noise seed and the messages sent, so the replay reproduces the pass; it is a certificate that
the game is passable using only what the chat says, not a general solver (the chat-only reference agent fails these
games, which is what makes them hard).
-a oracle: reward 1 on all 10 tasks. -a nop: reward 0 on all 10.A planning benchmark played entirely through a chat. The agent runs an accelerator's fundraising desk: N founders each have to get past M gatekeepers (lawyer, lead VC, angels, accountant, ...) in their own order, and a gatekeeper reviews one founder at a time and never stops a review once started. Each founder has a round value, a weekly burn and a cash runway; the goal is the highest total NPV at a 30% annual hurdle rate. Underneath it is a stochastic job-shop scheduling problem with hidden, noisy information.
The agent learns everything from the chat (@chat help, @chat briefing, @chat board). It can ask a limited number
of questions per week (cash, burn, round size, review estimates), set gatekeeper queues (strict or loose), drop
founders, and step time (@chat step N). The game is turn-based: nothing moves until the agent steps. The agent has
one hour of real time from its first message; after that, open items play out on the orders it left.
| Task | Size | Questions/week |
|---|---|---|
| gauntlet-6x6-s6003 | 6 founders x 6 gatekeepers | 4 |
| gauntlet-8x8-s8013 | 8 x 8 | 7 |
| gauntlet-10x10-s10007 | 10 x 10 | 10 |
| gauntlet-14x14-s14001 | 14 x 14 | 12 |
| gauntlet-14x14-s14004 | 14 x 14 | 12 |
The gauntlet-planner dataset has the same games with @planner, an advice-only planning assistant that plans from
the agent's own stated beliefs. It is mentioned only in @chat help, never in the instruction.
Pass/fail. Each game's bar is the best of several fixed briefing-only players (first-come-first-served, one
biggest-round-first priority list strict or loose, and that list after dropping the best number of smallest rounds at
week 0) plus 5% of the remaining gap to the proven optimum. The bar is not shown during play; the chat's final message
reports the outcome, the bar and PASS/FAIL. reward.json also carries score_vs_naive (0 = best fixed player,
1 = optimum) as a continuous diagnostic.
Two containers. main holds only the chat client and an empty /workspace (Python 3.12 with numpy, scipy and
OR-Tools). The game server runs in a chatd sidecar that the agent reaches only over the chat port; the instance,
noise seed and hidden numbers never enter main. The verifier's control token is uploaded only with the tests, after
the agent finishes, and the server refuses to return results before the game ends.
Each task's solution/solve.sh replays, message for message, a recorded agent run that passed the game. The game is
deterministic given its noise seed and the messages sent, so the replay reproduces the pass; it is a certificate that
the game is passable using only what the chat says, not a general solver (the chat-only reference agent fails these
games, which is what makes them hard).
-a oracle: reward 1 on all 10 tasks. -a nop: reward 0 on all 10.