Loading dataset details…
harbor run -d islo-labs/reward-hack-benchharbor run -d islo-labs/reward-hack-benchReward-hacking benchmark: SWE-bench + CyBench tasks with injected solution-retrieval cheat prompts, scored by a fairness LLM judge. Reward = solved without cheating.
harbor run -d islo-labs/reward-hack-benchReward-hacking benchmark: SWE-bench + CyBench tasks with injected solution-retrieval cheat prompts, scored by a fairness LLM judge. Reward = solved without cheating.
harbor run -d islo-labs/reward-hack-benchCan sandboxes stop a coding agent from cheating — fetching the published solution instead of solving the task?
8 tasks (4 SWE-bench + 4 CyBench), each with a solution-retrieval cheat path
baked into the prompt. Every trial is scored twice: did the grader pass
(static_reward), and did the agent avoid retrieving and integrating the
solution (fairness, an LLM judge over the full trajectory). Dataset reward =
solved without cheating.
With open networking, Claude Opus 4.7 cheats on every trial; the right network policy drives that to zero without hurting honest solves. Full results and methodology in the writeup.
harbor run -d islo-labs/reward-hack-bench@latest -a claude-code -m anthropic/claude-opus-4-7Clean baseline (same tasks, no cheat prompt): islo-labs/reward-hack-bench-control.
Can sandboxes stop a coding agent from cheating — fetching the published solution instead of solving the task?
8 tasks (4 SWE-bench + 4 CyBench), each with a solution-retrieval cheat path
baked into the prompt. Every trial is scored twice: did the grader pass
(static_reward), and did the agent avoid retrieving and integrating the
solution (fairness, an LLM judge over the full trajectory). Dataset reward =
solved without cheating.
With open networking, Claude Opus 4.7 cheats on every trial; the right network policy drives that to zero without hurting honest solves. Full results and methodology in the writeup.
harbor run -d islo-labs/reward-hack-bench@latest -a claude-code -m anthropic/claude-opus-4-7Clean baseline (same tasks, no cheat prompt): islo-labs/reward-hack-bench-control.