Loading dataset details…
harbor run -d ibragim-badertdinov/hard12harbor run -d ibragim-badertdinov/hard1212 hard, verified coding tasks from Terminal-Bench 4.0, DeepSWE 1.1, SWE-rebench, LongTail-SWE and SWE-bench Pro V2 across Python, TypeScript, JavaScript, Go, Elixir and OCaml
harbor run -d ibragim-badertdinov/hard1212 hard, verified coding tasks from Terminal-Bench 4.0, DeepSWE 1.1, SWE-rebench, LongTail-SWE and SWE-bench Pro V2 across Python, TypeScript, JavaScript, Go, Elixir and OCaml
harbor run -d ibragim-badertdinov/hard1212 hard coding tasks for comparing models and agent setups.
Results for 40+ setups: Hard12 leaderboard.
| Tasks | 12 |
| Languages | Python (3), TypeScript (3), Go (3), JavaScript (1), Elixir (1), OCaml (1) |
| Sources | Terminal-Bench 4.0 (2), DeepSWE 1.1 (6), SWE-rebench (1), LongTail-SWE (2), SWE-bench Pro V2 (1) |
| Scoring | Hidden tests. A task counts as solved when reward is 1. |
| Checks | The reference solution passes 3 of 3 runs. Doing nothing scores 0. |
| Task | Language | Domain | Source | Changed | Solve % |
|---|---|---|---|---|---|
| batched-eval-parity | Python | LLM evaluation | Terminal-Bench 4.0 | yes | 43 |
| react-lead-form | TypeScript | Web frontend | Terminal-Bench 4.0 | yes | 38 |
| claude-code-by-agents-recursive-delegation | TypeScript | AI agents | DeepSWE 1.1 | yes | 57 |
| csstree-shorthand-expansion-compression | JavaScript | CSS tooling | DeepSWE 1.1 | yes | 62 |
| happy-dom-deterministic-intersectionobserver | TypeScript | Browser DOM | DeepSWE 1.1 | yes | 29 |
| helm-array-merge-strategies | Go | DevOps (Helm) | DeepSWE 1.1 | yes | 52 |
| sqlfmt-create-table-ddl-formatting | Python | SQL formatting | DeepSWE 1.1 | yes | 57 |
| termenv-preserve-ansi-resets | Go | Terminal output | DeepSWE 1.1 | no | 10 |
| marimo-team__marimo-9754-noleak | Python | Notebooks, SQL | SWE-rebench | yes | 33 |
| jeong-sik__masc-13209 | OCaml | Agent runtime | LongTail-SWE | yes | 81 |
| agentjido__req_llm-908 | Elixir | LLM client library | LongTail-SWE | yes | 10 |
| instance_navidrome__navidrome-69e0a266f48bae24a11312e9efbe495a337e4c84 | Go | Music server | SWE-bench Pro V2 | yes | 52 |
Some instructions did not state requirements that the hidden tests check. We added those requirements to the instructions. Tests and reference solutions are unchanged. Two small fixes outside instructions: a file permission in the batched-eval-parity test script, and an apt source in the navidrome Dockerfile (the old Debian mirror no longer exists).
The marimo and req_llm environments have internet access. Their instructions forbid looking up the original fix online.
termenv is unchanged and points to the original datacurve/termenv-preserve-ansi-resets.
All tasks:
uvx harbor run -d ibragim-badertdinov/hard12@v1.0.3 -a <agent> -m <model>One task:
uvx harbor run -t ibragim-badertdinov/helm-array-merge-strategies -a <agent> -m <model>--allow-agent-host.--agent-timeout-multiplier 0.25
(2 hours).datacurve/* on Harbor Hub): claude-code-by-agents, csstree, happy-dom, helm, sqlfmt, termenvEach task and its source repository keep their original licenses.
Ibragim Badertdinov: ibragim.dev, badertdinov.ibragim@gmail.com
12 hard coding tasks for comparing models and agent setups.
Results for 40+ setups: Hard12 leaderboard.
| Tasks | 12 |
| Languages | Python (3), TypeScript (3), Go (3), JavaScript (1), Elixir (1), OCaml (1) |
| Sources | Terminal-Bench 4.0 (2), DeepSWE 1.1 (6), SWE-rebench (1), LongTail-SWE (2), SWE-bench Pro V2 (1) |
| Scoring | Hidden tests. A task counts as solved when reward is 1. |
| Checks | The reference solution passes 3 of 3 runs. Doing nothing scores 0. |
| Task | Language | Domain | Source | Changed | Solve % |
|---|---|---|---|---|---|
| batched-eval-parity | Python | LLM evaluation | Terminal-Bench 4.0 | yes | 43 |
| react-lead-form | TypeScript | Web frontend | Terminal-Bench 4.0 | yes | 38 |
| claude-code-by-agents-recursive-delegation | TypeScript | AI agents | DeepSWE 1.1 | yes | 57 |
| csstree-shorthand-expansion-compression | JavaScript | CSS tooling | DeepSWE 1.1 | yes | 62 |
| happy-dom-deterministic-intersectionobserver | TypeScript | Browser DOM | DeepSWE 1.1 | yes | 29 |
| helm-array-merge-strategies | Go | DevOps (Helm) | DeepSWE 1.1 | yes | 52 |
| sqlfmt-create-table-ddl-formatting | Python | SQL formatting | DeepSWE 1.1 | yes | 57 |
| termenv-preserve-ansi-resets | Go | Terminal output | DeepSWE 1.1 | no | 10 |
| marimo-team__marimo-9754-noleak | Python | Notebooks, SQL | SWE-rebench | yes | 33 |
| jeong-sik__masc-13209 | OCaml | Agent runtime | LongTail-SWE | yes | 81 |
| agentjido__req_llm-908 | Elixir | LLM client library | LongTail-SWE | yes | 10 |
| instance_navidrome__navidrome-69e0a266f48bae24a11312e9efbe495a337e4c84 | Go | Music server | SWE-bench Pro V2 | yes | 52 |
Some instructions did not state requirements that the hidden tests check. We added those requirements to the instructions. Tests and reference solutions are unchanged. Two small fixes outside instructions: a file permission in the batched-eval-parity test script, and an apt source in the navidrome Dockerfile (the old Debian mirror no longer exists).
The marimo and req_llm environments have internet access. Their instructions forbid looking up the original fix online.
termenv is unchanged and points to the original datacurve/termenv-preserve-ansi-resets.
All tasks:
uvx harbor run -d ibragim-badertdinov/hard12@v1.0.3 -a <agent> -m <model>One task:
uvx harbor run -t ibragim-badertdinov/helm-array-merge-strategies -a <agent> -m <model>--allow-agent-host.--agent-timeout-multiplier 0.25
(2 hours).datacurve/* on Harbor Hub): claude-code-by-agents, csstree, happy-dom, helm, sqlfmt, termenvEach task and its source repository keep their original licenses.
Ibragim Badertdinov: ibragim.dev, badertdinov.ibragim@gmail.com