harbor run -d swe-bench/swe-bench-verifiedSWE-bench Verified
SWE-bench Verified is a software-engineering dataset for evaluating whether AI agents can resolve real GitHub issues. Each task gives an agent an issue report and a repository checkout at the relevant base commit; the agent edits the code, and the solution is graded by running issue-specific tests.
SWE-bench contains 2,294 tasks from real GitHub issues and pull requests across 12 popular Python repositories. SWE-bench Verified is a 500-task human-validated subset, created with OpenAI, that checks for clear issue statements, correct tests, and solvable tasks.
Links
Parity
Harbor parity records are in adapters/swebench/parity_experiment.json. Historical validation compares original SWE-bench Verified to the Terminal-Bench adapter, then the legacy Terminal-Bench adapter to the Harbor adapter.
| Comparison | Agent | Model | Tasks | Reference | Harbor/TB |
|---|---|---|---|---|---|
| Original vs. TB | codex@0.2.0 |
o4-mini |
500 | 53.11 | 53.11 |
| Original vs. TB | terminus-v1.5-xml |
claude-opus-4-20250514 |
500 | 66.4 | 66.4 |
| Original vs. TB | openhands@c677f728 |
claude-4-sonnet |
500 | 66.8 | 67.0 |
| TB vs. Harbor | terminus-2@0bad1ff4 |
Claude-Sonnet-4-5-20250929 |
500 | 70.0 | 68.6 |
The adapter also includes a full-run comparison against a Docent SWE-bench Verified leaderboard run for mini-swe-agent==2.1.0 with openai/gpt-5-mini: official 0.563 +/- 0.000 vs. Harbor 0.545 +/- 0.007 over 499 comparable tasks and 3 Harbor runs. The excluded task was scikit-learn__scikit-learn-14710, a recurring Daytona outlier. Uploaded record: https://huggingface.co/datasets/harborframework/parity-experiments/discussions/149.
Citation
@inproceedings{
jimenez2024swebench,
title={{SWE}-bench: Can Language Models Resolve Real-world Github Issues?},
author={Carlos E Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik R Narasimhan},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=VTF8yNQM66}
}| Task |
|---|
swe-bench/scikit-learn__scikit-learn-13779 |
swe-bench/matplotlib__matplotlib-23412 |
swe-bench/django__django-15569 |
swe-bench/sympy__sympy-22714 |
swe-bench/django__django-13195 |
swe-bench/sphinx-doc__sphinx-7440 |
swe-bench/django__django-13590 |
swe-bench/sympy__sympy-19346 |
swe-bench/sympy__sympy-16597 |
swe-bench/django__django-14349 |
swe-bench/django__django-15916 |
swe-bench/sympy__sympy-18199 |
swe-bench/scikit-learn__scikit-learn-13124 |
swe-bench/sphinx-doc__sphinx-7985 |
swe-bench/django__django-16662 |
swe-bench/django__django-11551 |
swe-bench/scikit-learn__scikit-learn-26194 |
swe-bench/scikit-learn__scikit-learn-13496 |
swe-bench/django__django-15503 |
swe-bench/astropy__astropy-7671 |
swe-bench/pytest-dev__pytest-5809 |
swe-bench/pydata__xarray-4094 |
swe-bench/django__django-12741 |
swe-bench/django__django-16595 |
swe-bench/django__django-16485 |
swe-bench/django__django-11276 |
swe-bench/astropy__astropy-14309 |
swe-bench/django__django-15987 |
swe-bench/django__django-12713 |
swe-bench/sphinx-doc__sphinx-9367 |
swe-bench/matplotlib__matplotlib-25122 |
swe-bench/django__django-13401 |
swe-bench/django__django-15278 |
swe-bench/scikit-learn__scikit-learn-12682 |
swe-bench/sympy__sympy-12096 |
swe-bench/django__django-11087 |
swe-bench/sphinx-doc__sphinx-9698 |
swe-bench/matplotlib__matplotlib-25960 |
swe-bench/pydata__xarray-7393 |
swe-bench/sympy__sympy-23824 |
swe-bench/pytest-dev__pytest-7982 |
swe-bench/scikit-learn__scikit-learn-26323 |
swe-bench/django__django-15499 |
swe-bench/django__django-11149 |
swe-bench/sympy__sympy-13615 |
swe-bench/django__django-14007 |
swe-bench/sympy__sympy-19040 |
swe-bench/django__django-13670 |
swe-bench/scikit-learn__scikit-learn-10844 |
swe-bench/scikit-learn__scikit-learn-25102 |
swe-bench/django__django-13028 |
swe-bench/pylint-dev__pylint-6903 |
swe-bench/pytest-dev__pytest-8399 |
swe-bench/django__django-13344 |
swe-bench/sphinx-doc__sphinx-9461 |
swe-bench/django__django-12276 |
swe-bench/django__django-16333 |
swe-bench/sympy__sympy-13974 |
swe-bench/django__django-11603 |
swe-bench/django__django-12663 |
swe-bench/pytest-dev__pytest-6197 |
swe-bench/django__django-12754 |
swe-bench/django__django-11211 |
swe-bench/scikit-learn__scikit-learn-25973 |
swe-bench/django__django-16661 |
swe-bench/django__django-14725 |
swe-bench/matplotlib__matplotlib-23314 |
swe-bench/django__django-11964 |
swe-bench/sympy__sympy-19783 |
swe-bench/sympy__sympy-13091 |
swe-bench/django__django-15382 |
swe-bench/django__django-14238 |
swe-bench/matplotlib__matplotlib-20488 |
swe-bench/django__django-11451 |
swe-bench/django__django-16256 |
swe-bench/matplotlib__matplotlib-24149 |
swe-bench/sympy__sympy-24661 |
swe-bench/django__django-15525 |
swe-bench/pydata__xarray-4075 |
swe-bench/sphinx-doc__sphinx-9320 |
swe-bench/sphinx-doc__sphinx-8459 |
swe-bench/pydata__xarray-4687 |
swe-bench/django__django-15732 |
swe-bench/sympy__sympy-16792 |
swe-bench/sphinx-doc__sphinx-10673 |
swe-bench/sphinx-doc__sphinx-7454 |
swe-bench/sympy__sympy-24213 |
swe-bench/sphinx-doc__sphinx-10614 |
swe-bench/sphinx-doc__sphinx-8120 |
swe-bench/django__django-12774 |
swe-bench/django__django-14017 |
swe-bench/django__django-16819 |
swe-bench/scikit-learn__scikit-learn-14629 |
swe-bench/django__django-14752 |
swe-bench/django__django-11532 |
swe-bench/django__django-13821 |
swe-bench/django__django-15128 |
swe-bench/pytest-dev__pytest-7324 |
swe-bench/django__django-9296 |
swe-bench/django__django-14915 |
Displaying 100 of 500 tasks