ibragim-badertdinov/swe-rebench-v2.1-rl

harbor run -d ibragim-badertdinov/swe-rebench-v2.1-rl

SWE-rebench V2.1 RL

13,337 executable software-engineering tasks for RL training of coding agents, in Harbor format. Tasks come from SWE-rebench V2: real GitHub issues with the merged fix as the reference solution and the repository tests as the verifier.


Tasks	13,337
Languages	Python (3,990), Go (2,269), TypeScript (1,798), JavaScript (1,687), Rust (1,129), Java (573), Julia (434), PHP (424), Kotlin (306), Elixir (186), Scala (137), Swift (127), Dart (89), R (69), Clojure (53), C# (22), Lua (16), C (15), OCaml (11), C++ (2)
Environment	Prebuilt public image per task, pinned by digest
Scoring	Binary: reward 1 when all FAIL_TO_PASS tests pass and no PASS_TO_PASS test fails

Changes relative to SWE-rebench V2

Statements and tests were reviewed by LLM judges; defects were repaired or the task was removed.
Images were audited for leaked fixes and tests and rebuilt where needed.
Log parsers were fixed (Jest, Mocha, Julia, duplicate test names).
Tasks with a failing reference patch or an unstable environment were removed.

Task format

tests/test.sh applies tests/test.patch, runs the test command and grades the log with tests/grade.py.
tests/grade.py is a standalone standard-library grader; tests/config.json lists FAIL_TO_PASS and PASS_TO_PASS. Nothing is downloaded at verification time.
solution/solve.sh applies the reference patch.
[metadata] in task.toml holds repo, language and log_parser.

Run

uvx harbor run -d ibragim-badertdinov/swe-rebench-v2.1-rl -a <agent> -m <model>

Citation

If you use this dataset, please cite the SWE-rebench V2 paper:

@misc{badertdinov2026swerebenchv2languageagnosticswe,
      title={SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale},
      author={Ibragim Badertdinov and Maksim Nekrashevich and Anton Shevtsov and Alexander Golubev},
      year={2026},
      eprint={2602.23866},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2602.23866},
}