harbor run -d swe-bench/swe-bench-verifiedSWE-bench Verified
SWE-bench Verified is a software-engineering dataset for evaluating whether AI agents can resolve real GitHub issues. Each task gives an agent an issue report and a repository checkout at the relevant base commit; the agent edits the code, and the solution is graded by running issue-specific tests.
SWE-bench contains 2,294 tasks from real GitHub issues and pull requests across 12 popular Python repositories. SWE-bench Verified is a 500-task human-validated subset, created with OpenAI, that checks for clear issue statements, correct tests, and solvable tasks.
Links
Parity
Harbor parity records are in adapters/swebench/parity_experiment.json. Historical validation compares original SWE-bench Verified to the Terminal-Bench adapter, then the legacy Terminal-Bench adapter to the Harbor adapter.
| Comparison | Agent | Model | Tasks | Reference | Harbor/TB |
|---|---|---|---|---|---|
| Original vs. TB | codex@0.2.0 |
o4-mini |
500 | 53.11 | 53.11 |
| Original vs. TB | terminus-v1.5-xml |
claude-opus-4-20250514 |
500 | 66.4 | 66.4 |
| Original vs. TB | openhands@c677f728 |
claude-4-sonnet |
500 | 66.8 | 67.0 |
| TB vs. Harbor | terminus-2@0bad1ff4 |
Claude-Sonnet-4-5-20250929 |
500 | 70.0 | 68.6 |
The adapter also includes a full-run comparison against a Docent SWE-bench Verified leaderboard run for mini-swe-agent==2.1.0 with openai/gpt-5-mini: official 0.563 +/- 0.000 vs. Harbor 0.545 +/- 0.007 over 499 comparable tasks and 3 Harbor runs. The excluded task was scikit-learn__scikit-learn-14710, a recurring Daytona outlier. Uploaded record: https://huggingface.co/datasets/harborframework/parity-experiments/discussions/149.
Citation
@inproceedings{
jimenez2024swebench,
title={{SWE}-bench: Can Language Models Resolve Real-world Github Issues?},
author={Carlos E Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik R Narasimhan},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=VTF8yNQM66}
}| Task |
|---|
swe-bench/sphinx-doc__sphinx-11445 |
swe-bench/astropy__astropy-13033 |
swe-bench/django__django-15731 |
swe-bench/django__django-11749 |
swe-bench/astropy__astropy-14539 |
swe-bench/pydata__xarray-7233 |
swe-bench/matplotlib__matplotlib-25311 |
swe-bench/django__django-13343 |
swe-bench/django__django-16560 |
swe-bench/sympy__sympy-21930 |
swe-bench/django__django-12304 |
swe-bench/django__django-16569 |
swe-bench/django__django-14608 |
swe-bench/sphinx-doc__sphinx-7910 |
swe-bench/matplotlib__matplotlib-24570 |
swe-bench/django__django-13363 |
swe-bench/django__django-13410 |
swe-bench/django__django-11292 |
swe-bench/django__django-12039 |
swe-bench/django__django-12308 |
swe-bench/django__django-16100 |
swe-bench/django__django-14559 |
swe-bench/pydata__xarray-6992 |
swe-bench/django__django-14404 |
swe-bench/sphinx-doc__sphinx-7748 |
swe-bench/mwaskom__seaborn-3187 |
swe-bench/sympy__sympy-14248 |
swe-bench/psf__requests-1766 |
swe-bench/django__django-15368 |
swe-bench/django__django-14765 |
swe-bench/sympy__sympy-13480 |
swe-bench/sympy__sympy-21847 |
swe-bench/sympy__sympy-15349 |
swe-bench/django__django-13516 |
swe-bench/django__django-13089 |
swe-bench/django__django-12193 |
swe-bench/django__django-14493 |
swe-bench/django__django-15375 |
swe-bench/django__django-17087 |
swe-bench/sphinx-doc__sphinx-8595 |
swe-bench/scikit-learn__scikit-learn-25232 |
swe-bench/astropy__astropy-7336 |
swe-bench/sphinx-doc__sphinx-9658 |
swe-bench/django__django-14580 |
swe-bench/sphinx-doc__sphinx-10449 |
swe-bench/pytest-dev__pytest-10081 |
swe-bench/sympy__sympy-14976 |
swe-bench/sphinx-doc__sphinx-7462 |
swe-bench/django__django-14500 |
swe-bench/django__django-13158 |
swe-bench/django__django-13128 |
swe-bench/sympy__sympy-18189 |
swe-bench/sphinx-doc__sphinx-10323 |
swe-bench/pytest-dev__pytest-7205 |
swe-bench/sphinx-doc__sphinx-7889 |
swe-bench/sympy__sympy-19495 |
swe-bench/django__django-14787 |
swe-bench/django__django-14373 |
swe-bench/matplotlib__matplotlib-26466 |
swe-bench/django__django-16642 |
swe-bench/pytest-dev__pytest-10051 |
swe-bench/sphinx-doc__sphinx-8056 |
swe-bench/matplotlib__matplotlib-24177 |
swe-bench/django__django-11790 |
swe-bench/matplotlib__matplotlib-24026 |
swe-bench/django__django-14855 |
swe-bench/sphinx-doc__sphinx-8475 |
swe-bench/django__django-16255 |
swe-bench/sphinx-doc__sphinx-8548 |
swe-bench/scikit-learn__scikit-learn-13328 |
swe-bench/django__django-11880 |
swe-bench/django__django-11400 |
swe-bench/matplotlib__matplotlib-25775 |
swe-bench/django__django-13112 |
swe-bench/sphinx-doc__sphinx-9591 |
swe-bench/django__django-16527 |
swe-bench/django__django-12273 |
swe-bench/scikit-learn__scikit-learn-14710 |
swe-bench/django__django-13512 |
swe-bench/sympy__sympy-15875 |
swe-bench/django__django-10097 |
swe-bench/django__django-15629 |
swe-bench/django__django-16631 |
swe-bench/pydata__xarray-6721 |
swe-bench/sympy__sympy-14711 |
swe-bench/django__django-16315 |
swe-bench/django__django-15380 |
swe-bench/django__django-10973 |
swe-bench/sympy__sympy-24562 |
swe-bench/sphinx-doc__sphinx-10435 |
swe-bench/django__django-16493 |
swe-bench/scikit-learn__scikit-learn-12585 |
swe-bench/scikit-learn__scikit-learn-13135 |
swe-bench/django__django-11815 |
swe-bench/sphinx-doc__sphinx-8265 |
swe-bench/django__django-16667 |
swe-bench/django__django-15863 |
swe-bench/pydata__xarray-6938 |
swe-bench/psf__requests-6028 |
swe-bench/pydata__xarray-4629 |
Displaying 100 of 500 tasks