harbor run -d swe-bench/swe-bench-verifiedSWE-bench Verified
SWE-bench Verified is a software-engineering dataset for evaluating whether AI agents can resolve real GitHub issues. Each task gives an agent an issue report and a repository checkout at the relevant base commit; the agent edits the code, and the solution is graded by running issue-specific tests.
SWE-bench contains 2,294 tasks from real GitHub issues and pull requests across 12 popular Python repositories. SWE-bench Verified is a 500-task human-validated subset, created with OpenAI, that checks for clear issue statements, correct tests, and solvable tasks.
Links
Parity
Harbor parity records are in adapters/swebench/parity_experiment.json. Historical validation compares original SWE-bench Verified to the Terminal-Bench adapter, then the legacy Terminal-Bench adapter to the Harbor adapter.
| Comparison | Agent | Model | Tasks | Reference | Harbor/TB |
|---|---|---|---|---|---|
| Original vs. TB | codex@0.2.0 |
o4-mini |
500 | 53.11 | 53.11 |
| Original vs. TB | terminus-v1.5-xml |
claude-opus-4-20250514 |
500 | 66.4 | 66.4 |
| Original vs. TB | openhands@c677f728 |
claude-4-sonnet |
500 | 66.8 | 67.0 |
| TB vs. Harbor | terminus-2@0bad1ff4 |
Claude-Sonnet-4-5-20250929 |
500 | 70.0 | 68.6 |
The adapter also includes a full-run comparison against a Docent SWE-bench Verified leaderboard run for mini-swe-agent==2.1.0 with openai/gpt-5-mini: official 0.563 +/- 0.000 vs. Harbor 0.545 +/- 0.007 over 499 comparable tasks and 3 Harbor runs. The excluded task was scikit-learn__scikit-learn-14710, a recurring Daytona outlier. Uploaded record: https://huggingface.co/datasets/harborframework/parity-experiments/discussions/149.
Citation
@inproceedings{
jimenez2024swebench,
title={{SWE}-bench: Can Language Models Resolve Real-world Github Issues?},
author={Carlos E Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik R Narasimhan},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=VTF8yNQM66}
}| Task |
|---|
swe-bench/matplotlib__matplotlib-14623 |
swe-bench/django__django-16938 |
swe-bench/sympy__sympy-16886 |
swe-bench/psf__requests-1142 |
swe-bench/django__django-10554 |
swe-bench/django__django-16950 |
swe-bench/django__django-16032 |
swe-bench/pytest-dev__pytest-7236 |
swe-bench/django__django-14311 |
swe-bench/matplotlib__matplotlib-22865 |
swe-bench/django__django-16901 |
swe-bench/django__django-15973 |
swe-bench/matplotlib__matplotlib-25479 |
swe-bench/matplotlib__matplotlib-26208 |
swe-bench/sympy__sympy-24539 |
swe-bench/django__django-13121 |
swe-bench/astropy__astropy-8707 |
swe-bench/django__django-14631 |
swe-bench/django__django-15022 |
swe-bench/django__django-14999 |
swe-bench/django__django-13346 |
swe-bench/sympy__sympy-12489 |
swe-bench/astropy__astropy-14995 |
swe-bench/django__django-14140 |
swe-bench/django__django-16116 |
swe-bench/django__django-15037 |
swe-bench/django__django-11477 |
swe-bench/sympy__sympy-20590 |
swe-bench/sphinx-doc__sphinx-9230 |
swe-bench/scikit-learn__scikit-learn-15100 |
swe-bench/django__django-7530 |
swe-bench/matplotlib__matplotlib-22871 |
swe-bench/sympy__sympy-23262 |
swe-bench/pytest-dev__pytest-7432 |
swe-bench/django__django-13212 |
swe-bench/astropy__astropy-14096 |
swe-bench/sympy__sympy-17655 |
swe-bench/django__django-15561 |
swe-bench/django__django-15563 |
swe-bench/django__django-14376 |
swe-bench/astropy__astropy-13579 |
swe-bench/django__django-11141 |
swe-bench/django__django-16612 |
swe-bench/django__django-15572 |
swe-bench/django__django-15127 |
swe-bench/pydata__xarray-3151 |
swe-bench/pydata__xarray-3305 |
swe-bench/sphinx-doc__sphinx-9602 |
swe-bench/django__django-12419 |
swe-bench/pylint-dev__pylint-4604 |
swe-bench/sympy__sympy-12481 |
swe-bench/django__django-11951 |
swe-bench/django__django-14034 |
swe-bench/django__django-12125 |
swe-bench/django__django-13417 |
swe-bench/sympy__sympy-17139 |
swe-bench/django__django-13023 |
swe-bench/django__django-16139 |
swe-bench/pytest-dev__pytest-5631 |
swe-bench/sympy__sympy-13647 |
swe-bench/psf__requests-2931 |
swe-bench/django__django-11490 |
swe-bench/sympy__sympy-13757 |
swe-bench/matplotlib__matplotlib-24970 |
swe-bench/pylint-dev__pylint-4661 |
swe-bench/pydata__xarray-4695 |
swe-bench/sympy__sympy-15345 |
swe-bench/django__django-11179 |
swe-bench/django__django-13406 |
swe-bench/scikit-learn__scikit-learn-14496 |
swe-bench/django__django-15851 |
swe-bench/matplotlib__matplotlib-26291 |
swe-bench/matplotlib__matplotlib-25332 |
swe-bench/django__django-13033 |
swe-bench/django__django-14434 |
swe-bench/pylint-dev__pylint-4551 |
swe-bench/psf__requests-2317 |
swe-bench/sphinx-doc__sphinx-11510 |
swe-bench/django__django-15741 |
swe-bench/sympy__sympy-20428 |
swe-bench/django__django-15280 |
swe-bench/sphinx-doc__sphinx-8638 |
swe-bench/astropy__astropy-14369 |
swe-bench/scikit-learn__scikit-learn-14053 |
swe-bench/astropy__astropy-13977 |
swe-bench/matplotlib__matplotlib-20676 |
swe-bench/django__django-17084 |
swe-bench/sympy__sympy-16450 |
swe-bench/django__django-11099 |
swe-bench/django__django-13933 |
swe-bench/django__django-14122 |
swe-bench/astropy__astropy-13398 |
swe-bench/django__django-16263 |
swe-bench/matplotlib__matplotlib-22719 |
swe-bench/pylint-dev__pylint-7277 |
swe-bench/pytest-dev__pytest-5840 |
swe-bench/django__django-12262 |
swe-bench/scikit-learn__scikit-learn-10297 |
swe-bench/django__django-13794 |
swe-bench/matplotlib__matplotlib-24637 |
Displaying 100 of 500 tasks