SWE-Bench Pro is a benchmark designed to provide a rigorous and realistic evaluation of AI agents for software engineering. It was developed to address several limitations in existing benchmarks by tackling four key challenges:

1. Data Contamination: Models have likely seen the evaluation code during training, making it hard to know if they are problem-solving or recalling a memorized solution.
2. Limited Task Diversity: Many benchmarks fail to capture the full spectrum of real-world software challenges and instead focus on simple utility libraries.
3. Oversimplified Problems: Ambiguous or underspecified issues are often removed from benchmarks, which doesn't reflect a real developer's workflow.
4. Unreliable and Irreproducible Testing: Inconsistent setups make it difficult to know if a solution truly works or if the environment is just configured incorrectly.

SWE-Bench Pro addresses these gaps by sourcing tasks from diverse and complex codebases, including consumer applications, B2B services, and developer tools. To reduce contamination risk, the public and held-out OSS subsets use strong copyleft licenses (e.g., GPL). The private subset consists of proprietary codebases from startup partners.

The benchmark is significantly more challenging than its predecessors; top models score around 23% on the SWE-Bench Pro public set, compared to 70%+ on SWE-Bench Verified. This provides a more accurate measure of an agent’s true problem-solving capabilities in environments that mirror professional software development.

Read the paper here: https://scale.com/research/swe_bench_pro

Check out the GitHub and view trajectories here: https://docent.transluce.org/dashboard/032fb63d-4992-4bfc-911d-3b7dafcb931f

(https://labs.scale.com/leaderboard/swe_bench_pro_public)

SWE-Bench Pro is a benchmark designed to provide a rigorous and realistic evaluation of AI agents for software engineering. It was developed to address several limitations in existing benchmarks by tackling four key challenges:

1. Data Contamination: Models have likely seen the evaluation code during training, making it hard to know if they are problem-solving or recalling a memorized solution.
2. Limited Task Diversity: Many benchmarks fail to capture the full spectrum of real-world software challenges and instead focus on simple utility libraries.
3. Oversimplified Problems: Ambiguous or underspecified issues are often removed from benchmarks, which doesn't reflect a real developer's workflow.
4. Unreliable and Irreproducible Testing: Inconsistent setups make it difficult to know if a solution truly works or if the environment is just configured incorrectly.

SWE-Bench Pro addresses these gaps by sourcing tasks from diverse and complex codebases, including consumer applications, B2B services, and developer tools. To reduce contamination risk, the public and held-out OSS subsets use strong copyleft licenses (e.g., GPL). The private subset consists of proprietary codebases from startup partners.

The benchmark is significantly more challenging than its predecessors; top models score around 23% on the SWE-Bench Pro public set, compared to 70%+ on SWE-Bench Verified. This provides a more accurate measure of an agent’s true problem-solving capabilities in environments that mirror professional software development.

Read the paper here: https://scale.com/research/swe_bench_pro

Check out the GitHub and view trajectories here: https://docent.transluce.org/dashboard/032fb63d-4992-4bfc-911d-3b7dafcb931f

(https://labs.scale.com/leaderboard/swe_bench_pro_public)