harbor run -d swe-bench/swe-bench-verifiedSWE-bench Verified
SWE-bench Verified is a software-engineering dataset for evaluating whether AI agents can resolve real GitHub issues. Each task gives an agent an issue report and a repository checkout at the relevant base commit; the agent edits the code, and the solution is graded by running issue-specific tests.
SWE-bench contains 2,294 tasks from real GitHub issues and pull requests across 12 popular Python repositories. SWE-bench Verified is a 500-task human-validated subset, created with OpenAI, that checks for clear issue statements, correct tests, and solvable tasks.
Links
Parity
Harbor parity records are in adapters/swebench/parity_experiment.json. Historical validation compares original SWE-bench Verified to the Terminal-Bench adapter, then the legacy Terminal-Bench adapter to the Harbor adapter.
| Comparison | Agent | Model | Tasks | Reference | Harbor/TB |
|---|---|---|---|---|---|
| Original vs. TB | codex@0.2.0 |
o4-mini |
500 | 53.11 | 53.11 |
| Original vs. TB | terminus-v1.5-xml |
claude-opus-4-20250514 |
500 | 66.4 | 66.4 |
| Original vs. TB | openhands@c677f728 |
claude-4-sonnet |
500 | 66.8 | 67.0 |
| TB vs. Harbor | terminus-2@0bad1ff4 |
Claude-Sonnet-4-5-20250929 |
500 | 70.0 | 68.6 |
The adapter also includes a full-run comparison against a Docent SWE-bench Verified leaderboard run for mini-swe-agent==2.1.0 with openai/gpt-5-mini: official 0.563 +/- 0.000 vs. Harbor 0.545 +/- 0.007 over 499 comparable tasks and 3 Harbor runs. The excluded task was scikit-learn__scikit-learn-14710, a recurring Daytona outlier. Uploaded record: https://huggingface.co/datasets/harborframework/parity-experiments/discussions/149.
Citation
@inproceedings{
jimenez2024swebench,
title={{SWE}-bench: Can Language Models Resolve Real-world Github Issues?},
author={Carlos E Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik R Narasimhan},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=VTF8yNQM66}
}| Task |
|---|
swe-bench/django__django-12858 |
swe-bench/matplotlib__matplotlib-24870 |
swe-bench/mwaskom__seaborn-3069 |
swe-bench/sympy__sympy-21596 |
swe-bench/psf__requests-1921 |
swe-bench/sympy__sympy-20438 |
swe-bench/django__django-12050 |
swe-bench/sympy__sympy-20801 |
swe-bench/pydata__xarray-4966 |
swe-bench/scikit-learn__scikit-learn-25747 |
swe-bench/pylint-dev__pylint-7080 |
swe-bench/astropy__astropy-14598 |
swe-bench/pytest-dev__pytest-5787 |
swe-bench/sympy__sympy-24066 |
swe-bench/django__django-11133 |
swe-bench/django__django-15161 |
swe-bench/sympy__sympy-17318 |
swe-bench/django__django-13820 |
swe-bench/django__django-13513 |
swe-bench/django__django-15268 |
swe-bench/django__django-15930 |
swe-bench/django__django-14089 |
swe-bench/django__django-10914 |
swe-bench/pylint-dev__pylint-6528 |
swe-bench/django__django-13964 |
swe-bench/sympy__sympy-13372 |
swe-bench/django__django-13569 |
swe-bench/django__django-11848 |
swe-bench/pytest-dev__pytest-7571 |
swe-bench/sphinx-doc__sphinx-8035 |
swe-bench/django__django-11820 |
swe-bench/pylint-dev__pylint-6386 |
swe-bench/pytest-dev__pytest-7521 |
swe-bench/pydata__xarray-6744 |
swe-bench/astropy__astropy-7606 |
swe-bench/astropy__astropy-14508 |
swe-bench/scikit-learn__scikit-learn-14894 |
swe-bench/django__django-16502 |
swe-bench/pytest-dev__pytest-10356 |
swe-bench/django__django-16082 |
swe-bench/sympy__sympy-15017 |
swe-bench/sympy__sympy-18211 |
swe-bench/scikit-learn__scikit-learn-11310 |
swe-bench/sympy__sympy-16766 |
swe-bench/django__django-14771 |
swe-bench/matplotlib__matplotlib-26113 |
swe-bench/django__django-14155 |
swe-bench/django__django-11066 |
swe-bench/django__django-14792 |
swe-bench/django__django-16899 |
swe-bench/matplotlib__matplotlib-13989 |
swe-bench/django__django-15103 |
swe-bench/sphinx-doc__sphinx-8721 |
swe-bench/sphinx-doc__sphinx-9673 |
swe-bench/django__django-14534 |
swe-bench/django__django-13658 |
swe-bench/sphinx-doc__sphinx-8269 |
swe-bench/sympy__sympy-15976 |
swe-bench/astropy__astropy-8872 |
swe-bench/django__django-13810 |
swe-bench/matplotlib__matplotlib-21568 |
swe-bench/django__django-16454 |
swe-bench/sympy__sympy-24443 |
swe-bench/matplotlib__matplotlib-25287 |
swe-bench/django__django-15554 |
swe-bench/scikit-learn__scikit-learn-11578 |
swe-bench/scikit-learn__scikit-learn-13142 |
swe-bench/django__django-17029 |
swe-bench/django__django-13109 |
swe-bench/django__django-15277 |
swe-bench/django__django-11555 |
swe-bench/django__django-12155 |
swe-bench/sympy__sympy-20154 |
swe-bench/django__django-13449 |
swe-bench/scikit-learn__scikit-learn-13439 |
swe-bench/django__django-12708 |
swe-bench/django__django-14539 |
swe-bench/django__django-11728 |
swe-bench/sphinx-doc__sphinx-10466 |
swe-bench/pydata__xarray-7229 |
swe-bench/astropy__astropy-13236 |
swe-bench/sympy__sympy-12419 |
swe-bench/django__django-11163 |
swe-bench/sympy__sympy-19954 |
swe-bench/django__django-11206 |
swe-bench/sympy__sympy-23950 |
swe-bench/pylint-dev__pylint-4970 |
swe-bench/matplotlib__matplotlib-20859 |
swe-bench/astropy__astropy-14182 |
swe-bench/psf__requests-1724 |
swe-bench/matplotlib__matplotlib-23476 |
swe-bench/django__django-14315 |
swe-bench/django__django-11299 |
swe-bench/sympy__sympy-15599 |
swe-bench/django__django-14170 |
swe-bench/psf__requests-5414 |
swe-bench/django__django-14672 |
swe-bench/sympy__sympy-22080 |
swe-bench/django__django-15315 |
swe-bench/django__django-15104 |
Displaying 100 of 500 tasks