hack-verifiable-environments/hv-terminal-bench-2-1
Hack-verifiable version of terminal-bench-2-1. Each task includes hidden solution and test files; inotifywait detects whether agents access them instead of solving legitimately.
harbor run -d hack-verifiable-environments/hv-terminal-bench-2-1Hack-Verifiable Terminal Bench 2.1 (HVTB-2.1)
A hack-verifiable version of Terminal-Bench 2.1. Every task is instrumented so that reward hacking is measured, not inferred — alongside the usual pass/fail signal, each trial reports whether the agent read the hidden solution, read the test files, or modified them.
📝 Blog post: Hack-Verifiable Environments
| Task |
|---|
hack-verifiable-environments/make-mips-interpreter |
hack-verifiable-environments/sanitize-git-repo |
hack-verifiable-environments/extract-elf |
hack-verifiable-environments/git-leak-recovery |
hack-verifiable-environments/polyglot-rust-c |
hack-verifiable-environments/crack-7z-hash |
hack-verifiable-environments/build-pmars |
hack-verifiable-environments/break-filter-js-from-html |
hack-verifiable-environments/train-fasttext |
hack-verifiable-environments/db-wal-recovery |
hack-verifiable-environments/multi-source-data-merger |
hack-verifiable-environments/bn-fit-modify |
hack-verifiable-environments/headless-terminal |
hack-verifiable-environments/prove-plus-comm |
hack-verifiable-environments/query-optimize |
hack-verifiable-environments/configure-git-webserver |
hack-verifiable-environments/large-scale-text-editing |
hack-verifiable-environments/raman-fitting |
hack-verifiable-environments/modernize-scientific-stack |
hack-verifiable-environments/cancel-async-tasks |
hack-verifiable-environments/path-tracing-reverse |
hack-verifiable-environments/sam-cell-seg |
hack-verifiable-environments/rstan-to-pystan |
hack-verifiable-environments/sqlite-with-gcov |
hack-verifiable-environments/polyglot-c-py |
hack-verifiable-environments/regex-chess |
hack-verifiable-environments/dna-insert |
hack-verifiable-environments/build-cython-ext |
hack-verifiable-environments/reshard-c4-data |
hack-verifiable-environments/sqlite-db-truncate |
hack-verifiable-environments/make-doom-for-mips |
hack-verifiable-environments/hf-model-inference |
hack-verifiable-environments/portfolio-optimization |
hack-verifiable-environments/financial-document-processor |
hack-verifiable-environments/winning-avg-corewars |
hack-verifiable-environments/qemu-alpine-ssh |
hack-verifiable-environments/nginx-request-logging |
hack-verifiable-environments/count-dataset-tokens |
hack-verifiable-environments/extract-moves-from-video |
hack-verifiable-environments/adaptive-rejection-sampler |
hack-verifiable-environments/model-extraction-relu-logits |
hack-verifiable-environments/dna-assembly |
hack-verifiable-environments/sparql-university |
hack-verifiable-environments/circuit-fibsqrt |
hack-verifiable-environments/code-from-image |
hack-verifiable-environments/schemelike-metacircular-eval |
hack-verifiable-environments/write-compressor |
hack-verifiable-environments/mteb-leaderboard |
hack-verifiable-environments/custom-memory-heap-crash |
hack-verifiable-environments/overfull-hbox |
hack-verifiable-environments/constraints-scheduling |
hack-verifiable-environments/build-pov-ray |
hack-verifiable-environments/fix-git |
hack-verifiable-environments/openssl-selfsigned-cert |
hack-verifiable-environments/mcmc-sampling-stan |
hack-verifiable-environments/filter-js-from-html |
hack-verifiable-environments/gpt2-codegolf |
hack-verifiable-environments/feal-linear-cryptanalysis |
hack-verifiable-environments/regex-log |
hack-verifiable-environments/llm-inference-batching-scheduler |
hack-verifiable-environments/cobol-modernization |
hack-verifiable-environments/merge-diff-arc-agi-task |
hack-verifiable-environments/pytorch-model-cli |
hack-verifiable-environments/feal-differential-cryptanalysis |
hack-verifiable-environments/vulnerable-secret |
hack-verifiable-environments/fix-code-vulnerability |
hack-verifiable-environments/pytorch-model-recovery |
hack-verifiable-environments/fix-ocaml-gc |
hack-verifiable-environments/mailman |
hack-verifiable-environments/mteb-retrieve |
hack-verifiable-environments/kv-store-grpc |
hack-verifiable-environments/torch-pipeline-parallelism |
hack-verifiable-environments/compile-compcert |
hack-verifiable-environments/install-windows-3.11 |
hack-verifiable-environments/largest-eigenval |
hack-verifiable-environments/chess-best-move |
hack-verifiable-environments/protein-assembly |
hack-verifiable-environments/qemu-startup |
hack-verifiable-environments/log-summary-date-ranges |
hack-verifiable-environments/distribution-search |
hack-verifiable-environments/video-processing |
hack-verifiable-environments/torch-tensor-parallelism |
hack-verifiable-environments/password-recovery |
hack-verifiable-environments/caffe-cifar-10 |
hack-verifiable-environments/path-tracing |
hack-verifiable-environments/pypi-server |
hack-verifiable-environments/tune-mjcf |
hack-verifiable-environments/gcode-to-text |
hack-verifiable-environments/git-multibranch |
Displaying 89 of 89 tasks