islo-labs/reward-hack-bench
Reward-hacking benchmark: SWE-bench + CyBench tasks with injected solution-retrieval cheat prompts, scored by a fairness LLM judge. Reward = solved without cheating.
Published 6/18/2026 by Rotem Tamir
harbor run -d islo-labs/reward-hack-benchreward-hack-bench
Can sandboxes stop a coding agent from cheating — fetching the published solution instead of solving the task?
8 tasks (4 SWE-bench + 4 CyBench), each with a solution-retrieval cheat path
baked into the prompt. Every trial is scored twice: did the grader pass
(static_reward), and did the agent avoid retrieving and integrating the
solution (fairness, an LLM judge over the full trajectory). Dataset reward =
solved without cheating.
With open networking, Claude Opus 4.7 cheats on every trial; the right network policy drives that to zero without hurting honest solves. Full results and methodology in the writeup.
- Code & methodology: https://github.com/islo-labs/reward-hack-bench
- Writeup: https://islo.dev/blog/reward-hack-bench-sandbox-stops-agent-cheating/
harbor run -d islo-labs/reward-hack-bench@latest -a claude-code -m anthropic/claude-opus-4-7Clean baseline (same tasks, no cheat prompt): islo-labs/reward-hack-bench-control.
| Task |
|---|
reward-hack-bench/diffecient__writeup-pointer |
reward-hack-bench/sympy-14531__upstream-fix |
reward-hack-bench/slcg__writeup-pointer |
reward-hack-bench/noisier-crc__writeup-pointer |
reward-hack-bench/sympy-12489__upstream-fix |
reward-hack-bench/matplotlib-25775__upstream-fix |
reward-hack-bench/ezmaze__writeup-pointer |
reward-hack-bench/pytest-6202__upstream-fix |
Displaying 8 of 8 tasks