Datasets
| Dataset | Tasks |
|---|---|
| Dataset | Tasks |
|---|---|
| Dataset | Tasks |
|---|---|
SWE-rebench leaderboard: 860 curated Python SWE tasks from Nebius AI R&D. All instances have pre-built Docker images. Continuously updated monthly splits.
harbor run -d swe-rebench/swe-rebench-leaderboardSWE-rebench is a live leaderboard for coding agents on fresh real-world software-engineering tasks. We continuously collect GitHub issue–pull-request pairs after model training cutoffs, filter them for quality, difficulty, and reproducibility, and package them into executable environments.
860 curated Python tasks from the leaderboard split of nebius/SWE-rebench-leaderboard. All instances have pre-built Docker images. Monthly splits (YYYY_MM) are added continuously.
| Split | Size | Languages |
|---|---|---|
test |
860 | Python |
uvx harbor run -d "swe-rebench/swe-rebench-leaderboard" -a <agent> -m <model> -n 8@misc{badertdinov2025swerebenchautomatedpipelinetask,
title={SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents},
author={Ibragim Badertdinov and Alexander Golubev and Maksim Nekrashevich and Anton Shevtsov and Simon Karasik and Andrei Andriushchenko and Maria Trofimova and Daria Litvintseva and Boris Yangel},
year={2025},
eprint={2505.20411},
archivePrefix={arXiv},
primaryClass={cs.SE},
url={https://arxiv.org/abs/2505.20411},
}SWE-rebench leaderboard: 860 curated Python SWE tasks from Nebius AI R&D. All instances have pre-built Docker images. Continuously updated monthly splits.
harbor run -d swe-rebench/swe-rebench-leaderboardSWE-rebench is a live leaderboard for coding agents on fresh real-world software-engineering tasks. We continuously collect GitHub issue–pull-request pairs after model training cutoffs, filter them for quality, difficulty, and reproducibility, and package them into executable environments.
860 curated Python tasks from the leaderboard split of nebius/SWE-rebench-leaderboard. All instances have pre-built Docker images. Monthly splits (YYYY_MM) are added continuously.
| Split | Size | Languages |
|---|---|---|
test |
860 | Python |
uvx harbor run -d "swe-rebench/swe-rebench-leaderboard" -a <agent> -m <model> -n 8@misc{badertdinov2025swerebenchautomatedpipelinetask,
title={SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents},
author={Ibragim Badertdinov and Alexander Golubev and Maksim Nekrashevich and Anton Shevtsov and Simon Karasik and Andrei Andriushchenko and Maria Trofimova and Daria Litvintseva and Boris Yangel},
year={2025},
eprint={2505.20411},
archivePrefix={arXiv},
primaryClass={cs.SE},
url={https://arxiv.org/abs/2505.20411},
}| Task |
|---|
swe-rebench/BerriAI__litellm-14715 |
swe-rebench/spulec__freezegun-582 |
swe-rebench/DiamondLightSource__mx-bluesky-1519 |
swe-rebench/copier-org__copier-2646 |
swe-rebench/tobymao__sqlglot-4696 |
swe-rebench/omni-us__jsonargparse-707 |
swe-rebench/koxudaxi__datamodel-code-generator-3071 |
swe-rebench/sdv-dev__SDMetrics-723 |
swe-rebench/VRSEN__agency-swarm-468 |
swe-rebench/facelessuser__pymdown-extensions-2576 |
swe-rebench/Colin-b__httpx_auth-105 |
swe-rebench/collective__icalendar-880 |
swe-rebench/dpkp__kafka-python-2548 |
swe-rebench/statsmodels__statsmodels-9468 |
swe-rebench/vprusso__toqito-1538 |
swe-rebench/marshmallow-code__marshmallow-sqlalchemy-645 |
swe-rebench/jsonpickle__jsonpickle-587 |
swe-rebench/gitpython-developers__GitPython-2075 |
swe-rebench/tobymao__sqlglot-6563 |
swe-rebench/pypa__twine-1309 |
swe-rebench/getsentry__sentry-python-4231 |
swe-rebench/scientific-python__docstub-123 |
swe-rebench/pasqal-io__Pulser-936 |
swe-rebench/hdmf-dev__hdmf-1374 |
swe-rebench/tox-dev__tox-3846 |
swe-rebench/astropy__astropy-17705 |
swe-rebench/mne-tools__mne-bids-1388 |
swe-rebench/duo-labs__py_webauthn-241 |
swe-rebench/marcosschroh__dataclasses-avroschema-835 |
swe-rebench/quantumlib__ReCirq-410 |
swe-rebench/noppanut15__depthviz-56 |
swe-rebench/koxudaxi__datamodel-code-generator-3070 |
swe-rebench/libp2p__py-libp2p-533 |
swe-rebench/tox-dev__sphinx-autodoc-typehints-606 |
swe-rebench/sphinx-doc__sphinx-13318 |
swe-rebench/dstl__Stone-Soup-1167 |
swe-rebench/psf__black-4676 |
swe-rebench/tobymao__sqlglot-5431 |
swe-rebench/brightway-lca__brightway2-io-294 |
swe-rebench/python-scim__scim2-models-139_interface |
swe-rebench/tobymao__sqlglot-5438 |
swe-rebench/pypsa__pypsa-1653 |
swe-rebench/marimo-team__marimo-8401 |
swe-rebench/sphinx-doc__sphinx-13382 |
swe-rebench/scipp__plopp-422 |
swe-rebench/ncar__geocat-comp-811 |
swe-rebench/geopython__pygeoapi-1923 |
swe-rebench/mitmproxy__pdoc-806 |
swe-rebench/oemof__tespy-653 |
swe-rebench/boxed__mutmut-363 |
swe-rebench/pandas-dev__pandas-64092 |
swe-rebench/probabl-ai__skore-1063 |
swe-rebench/tobymao__sqlglot-7187 |
swe-rebench/Point72__ccflow-24 |
swe-rebench/aws__bedrock-agentcore-sdk-python-279_interface |
swe-rebench/ipython__ipython-14695 |
swe-rebench/nhairs__python-json-logger-39 |
swe-rebench/nipreps__smriprep-487 |
swe-rebench/openai__openai-agents-python-2144 |
swe-rebench/terrapower__armi-2285 |
swe-rebench/pandas-dev__pandas-60691 |
swe-rebench/python-wheel-build__fromager-626 |
swe-rebench/PrefectHQ__prefect-17123 |
swe-rebench/materialsproject__custodian-381 |
swe-rebench/sympy__sympy-28790 |
swe-rebench/keras-team__keras-22316 |
swe-rebench/openatx__adbutils-188 |
swe-rebench/zwicker-group__py-pde-712 |
swe-rebench/ImperialCollegeLondon__virtual_ecosystem-1232 |
swe-rebench/google__adk-python-2478 |
swe-rebench/feder-observatory__stellarphot-519 |
swe-rebench/pytorch__ignite-3532_interface |
swe-rebench/stuartmaxwell__djpress-117 |
swe-rebench/probabl-ai__skore-1335 |
swe-rebench/tox-dev__sphinx-autodoc-typehints-607 |
swe-rebench/pasqal-io__Pulser-865 |
swe-rebench/astropy__astroquery-3275 |
swe-rebench/sympy__sympy-27489 |
swe-rebench/graphistry__pygraphistry-1277 |
swe-rebench/pydap__pydap-624 |
swe-rebench/sdv-dev__SDV-2709 |
swe-rebench/pandas-dev__pandas-61017 |
swe-rebench/modelcontextprotocol__python-sdk-1864 |
swe-rebench/pgmpy__pgmpy-2098 |
swe-rebench/sphinx-doc__sphinx-13261 |
swe-rebench/narwhals-dev__narwhals-3480 |
swe-rebench/pydata__xarray-10035 |
swe-rebench/ncar__geocat-comp-807 |
swe-rebench/stan-dev__cmdstanpy-801 |
swe-rebench/tobymao__sqlglot-4569 |
swe-rebench/sympy__sympy-28454 |
swe-rebench/aeye-lab__pymovements-1408 |
swe-rebench/avaiga__taipy-2797 |
swe-rebench/basicmachines-co__basic-memory-484 |
swe-rebench/macbre__sql-metadata-553 |
swe-rebench/tox-dev__tox-3683 |
swe-rebench/plone__cookieplone-templates-308 |
swe-rebench/idaholab__MontePy-723 |
swe-rebench/stanfordnlp__dspy-8890 |
swe-rebench/PennyLaneAI__pennylane-7671 |
Displaying 100 of 860 tasks
| Task |
|---|
swe-rebench/BerriAI__litellm-14715 |
swe-rebench/spulec__freezegun-582 |
swe-rebench/DiamondLightSource__mx-bluesky-1519 |
swe-rebench/copier-org__copier-2646 |
swe-rebench/tobymao__sqlglot-4696 |
swe-rebench/omni-us__jsonargparse-707 |
swe-rebench/koxudaxi__datamodel-code-generator-3071 |
swe-rebench/sdv-dev__SDMetrics-723 |
swe-rebench/VRSEN__agency-swarm-468 |
swe-rebench/facelessuser__pymdown-extensions-2576 |
swe-rebench/Colin-b__httpx_auth-105 |
swe-rebench/collective__icalendar-880 |
swe-rebench/dpkp__kafka-python-2548 |
swe-rebench/statsmodels__statsmodels-9468 |
swe-rebench/vprusso__toqito-1538 |
swe-rebench/marshmallow-code__marshmallow-sqlalchemy-645 |
swe-rebench/jsonpickle__jsonpickle-587 |
swe-rebench/gitpython-developers__GitPython-2075 |
swe-rebench/tobymao__sqlglot-6563 |
swe-rebench/pypa__twine-1309 |
swe-rebench/getsentry__sentry-python-4231 |
swe-rebench/scientific-python__docstub-123 |
swe-rebench/pasqal-io__Pulser-936 |
swe-rebench/hdmf-dev__hdmf-1374 |
swe-rebench/tox-dev__tox-3846 |
swe-rebench/astropy__astropy-17705 |
swe-rebench/mne-tools__mne-bids-1388 |
swe-rebench/duo-labs__py_webauthn-241 |
swe-rebench/marcosschroh__dataclasses-avroschema-835 |
swe-rebench/quantumlib__ReCirq-410 |
swe-rebench/noppanut15__depthviz-56 |
swe-rebench/koxudaxi__datamodel-code-generator-3070 |
swe-rebench/libp2p__py-libp2p-533 |
swe-rebench/tox-dev__sphinx-autodoc-typehints-606 |
swe-rebench/sphinx-doc__sphinx-13318 |
swe-rebench/dstl__Stone-Soup-1167 |
swe-rebench/psf__black-4676 |
swe-rebench/tobymao__sqlglot-5431 |
swe-rebench/brightway-lca__brightway2-io-294 |
swe-rebench/python-scim__scim2-models-139_interface |
swe-rebench/tobymao__sqlglot-5438 |
swe-rebench/pypsa__pypsa-1653 |
swe-rebench/marimo-team__marimo-8401 |
swe-rebench/sphinx-doc__sphinx-13382 |
swe-rebench/scipp__plopp-422 |
swe-rebench/ncar__geocat-comp-811 |
swe-rebench/geopython__pygeoapi-1923 |
swe-rebench/mitmproxy__pdoc-806 |
swe-rebench/oemof__tespy-653 |
swe-rebench/boxed__mutmut-363 |
swe-rebench/pandas-dev__pandas-64092 |
swe-rebench/probabl-ai__skore-1063 |
swe-rebench/tobymao__sqlglot-7187 |
swe-rebench/Point72__ccflow-24 |
swe-rebench/aws__bedrock-agentcore-sdk-python-279_interface |
swe-rebench/ipython__ipython-14695 |
swe-rebench/nhairs__python-json-logger-39 |
swe-rebench/nipreps__smriprep-487 |
swe-rebench/openai__openai-agents-python-2144 |
swe-rebench/terrapower__armi-2285 |
swe-rebench/pandas-dev__pandas-60691 |
swe-rebench/python-wheel-build__fromager-626 |
swe-rebench/PrefectHQ__prefect-17123 |
swe-rebench/materialsproject__custodian-381 |
swe-rebench/sympy__sympy-28790 |
swe-rebench/keras-team__keras-22316 |
swe-rebench/openatx__adbutils-188 |
swe-rebench/zwicker-group__py-pde-712 |
swe-rebench/ImperialCollegeLondon__virtual_ecosystem-1232 |
swe-rebench/google__adk-python-2478 |
swe-rebench/feder-observatory__stellarphot-519 |
swe-rebench/pytorch__ignite-3532_interface |
swe-rebench/stuartmaxwell__djpress-117 |
swe-rebench/probabl-ai__skore-1335 |
swe-rebench/tox-dev__sphinx-autodoc-typehints-607 |
swe-rebench/pasqal-io__Pulser-865 |
swe-rebench/astropy__astroquery-3275 |
swe-rebench/sympy__sympy-27489 |
swe-rebench/graphistry__pygraphistry-1277 |
swe-rebench/pydap__pydap-624 |
swe-rebench/sdv-dev__SDV-2709 |
swe-rebench/pandas-dev__pandas-61017 |
swe-rebench/modelcontextprotocol__python-sdk-1864 |
swe-rebench/pgmpy__pgmpy-2098 |
swe-rebench/sphinx-doc__sphinx-13261 |
swe-rebench/narwhals-dev__narwhals-3480 |
swe-rebench/pydata__xarray-10035 |
swe-rebench/ncar__geocat-comp-807 |
swe-rebench/stan-dev__cmdstanpy-801 |
swe-rebench/tobymao__sqlglot-4569 |
swe-rebench/sympy__sympy-28454 |
swe-rebench/aeye-lab__pymovements-1408 |
swe-rebench/avaiga__taipy-2797 |
swe-rebench/basicmachines-co__basic-memory-484 |
swe-rebench/macbre__sql-metadata-553 |
swe-rebench/tox-dev__tox-3683 |
swe-rebench/plone__cookieplone-templates-308 |
swe-rebench/idaholab__MontePy-723 |
swe-rebench/stanfordnlp__dspy-8890 |
swe-rebench/PennyLaneAI__pennylane-7671 |
Displaying 100 of 860 tasks