harbor run -d swe-bench/swe-bench-verifiedSWE-bench Verified
SWE-bench Verified is a software-engineering dataset for evaluating whether AI agents can resolve real GitHub issues. Each task gives an agent an issue report and a repository checkout at the relevant base commit; the agent edits the code, and the solution is graded by running issue-specific tests.
SWE-bench contains 2,294 tasks from real GitHub issues and pull requests across 12 popular Python repositories. SWE-bench Verified is a 500-task human-validated subset, created with OpenAI, that checks for clear issue statements, correct tests, and solvable tasks.
Links
Parity
Harbor parity records are in adapters/swebench/parity_experiment.json. Historical validation compares original SWE-bench Verified to the Terminal-Bench adapter, then the legacy Terminal-Bench adapter to the Harbor adapter.
| Comparison | Agent | Model | Tasks | Reference | Harbor/TB |
|---|---|---|---|---|---|
| Original vs. TB | codex@0.2.0 |
o4-mini |
500 | 53.11 | 53.11 |
| Original vs. TB | terminus-v1.5-xml |
claude-opus-4-20250514 |
500 | 66.4 | 66.4 |
| Original vs. TB | openhands@c677f728 |
claude-4-sonnet |
500 | 66.8 | 67.0 |
| TB vs. Harbor | terminus-2@0bad1ff4 |
Claude-Sonnet-4-5-20250929 |
500 | 70.0 | 68.6 |
The adapter also includes a full-run comparison against a Docent SWE-bench Verified leaderboard run for mini-swe-agent==2.1.0 with openai/gpt-5-mini: official 0.563 +/- 0.000 vs. Harbor 0.545 +/- 0.007 over 499 comparable tasks and 3 Harbor runs. The excluded task was scikit-learn__scikit-learn-14710, a recurring Daytona outlier. Uploaded record: https://huggingface.co/datasets/harborframework/parity-experiments/discussions/149.
Citation
@inproceedings{
jimenez2024swebench,
title={{SWE}-bench: Can Language Models Resolve Real-world Github Issues?},
author={Carlos E Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik R Narasimhan},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=VTF8yNQM66}
}| Task |
|---|
swe-bench/sympy__sympy-13877 |
swe-bench/django__django-15957 |
swe-bench/pydata__xarray-3677 |
swe-bench/django__django-13925 |
swe-bench/django__django-12325 |
swe-bench/sphinx-doc__sphinx-9281 |
swe-bench/pallets__flask-5014 |
swe-bench/django__django-13551 |
swe-bench/django__django-12209 |
swe-bench/sympy__sympy-15809 |
swe-bench/sympy__sympy-13031 |
swe-bench/sphinx-doc__sphinx-8551 |
swe-bench/sympy__sympy-14531 |
swe-bench/pydata__xarray-2905 |
swe-bench/scikit-learn__scikit-learn-14141 |
swe-bench/django__django-11239 |
swe-bench/sympy__sympy-13852 |
swe-bench/django__django-13741 |
swe-bench/django__django-12965 |
swe-bench/sympy__sympy-22456 |
swe-bench/django__django-11885 |
swe-bench/scikit-learn__scikit-learn-9288 |
swe-bench/django__django-15252 |
swe-bench/sphinx-doc__sphinx-7590 |
swe-bench/matplotlib__matplotlib-23299 |
swe-bench/django__django-16877 |
swe-bench/django__django-11095 |
swe-bench/django__django-11265 |
swe-bench/pydata__xarray-6461 |
swe-bench/sympy__sympy-13551 |
swe-bench/django__django-16429 |
swe-bench/scikit-learn__scikit-learn-14983 |
swe-bench/pydata__xarray-3095 |
swe-bench/django__django-10999 |
swe-bench/django__django-16801 |
swe-bench/sympy__sympy-23413 |
swe-bench/django__django-11138 |
swe-bench/sympy__sympy-13878 |
swe-bench/django__django-15467 |
swe-bench/pydata__xarray-6599 |
swe-bench/pytest-dev__pytest-5262 |
swe-bench/astropy__astropy-12907 |
swe-bench/sympy__sympy-21379 |
swe-bench/pytest-dev__pytest-7490 |
swe-bench/django__django-15098 |
swe-bench/django__django-14351 |
swe-bench/django__django-12406 |
swe-bench/django__django-13837 |
swe-bench/astropy__astropy-13453 |
swe-bench/scikit-learn__scikit-learn-12973 |
swe-bench/django__django-14053 |
swe-bench/scikit-learn__scikit-learn-10908 |
swe-bench/django__django-15695 |
swe-bench/django__django-16145 |
swe-bench/django__django-15814 |
swe-bench/sympy__sympy-13798 |
swe-bench/pylint-dev__pylint-8898 |
swe-bench/sympy__sympy-11618 |
swe-bench/django__django-14011 |
swe-bench/django__django-12143 |
swe-bench/sympy__sympy-17630 |
swe-bench/sphinx-doc__sphinx-7757 |
swe-bench/sympy__sympy-23534 |
swe-bench/django__django-13315 |
swe-bench/django__django-10880 |
swe-bench/matplotlib__matplotlib-24627 |
swe-bench/django__django-13279 |
swe-bench/django__django-11119 |
swe-bench/scikit-learn__scikit-learn-14087 |
swe-bench/sympy__sympy-22914 |
swe-bench/sphinx-doc__sphinx-8621 |
swe-bench/sympy__sympy-21612 |
swe-bench/sphinx-doc__sphinx-9229 |
swe-bench/sympy__sympy-18698 |
swe-bench/django__django-13012 |
swe-bench/sympy__sympy-20916 |
swe-bench/matplotlib__matplotlib-26342 |
swe-bench/sphinx-doc__sphinx-9258 |
swe-bench/matplotlib__matplotlib-20826 |
swe-bench/sympy__sympy-19637 |
swe-bench/django__django-16136 |
swe-bench/pytest-dev__pytest-6202 |
swe-bench/django__django-11999 |
swe-bench/django__django-11734 |
swe-bench/scikit-learn__scikit-learn-25931 |
swe-bench/django__django-11333 |
swe-bench/sympy__sympy-18763 |
swe-bench/django__django-11740 |
swe-bench/django__django-13297 |
swe-bench/astropy__astropy-7166 |
swe-bench/django__django-13568 |
swe-bench/django__django-11433 |
swe-bench/astropy__astropy-14365 |
swe-bench/pydata__xarray-3993 |
swe-bench/pydata__xarray-4356 |
swe-bench/django__django-13809 |
swe-bench/django__django-13786 |
swe-bench/django__django-13807 |
swe-bench/sphinx-doc__sphinx-8593 |
swe-bench/sphinx-doc__sphinx-9711 |
Displaying 100 of 500 tasks