SWE-bench
SWE-bench is a benchmark platform designed to assess the capabilities of major language models to solve GitHub real software problems. The system expects the model to create a patch that produces a solution on a given codebase and problem notification. Evaluation processes are carried out using a Docker-based container structure to achieve repeatable results.
SWE-bench · AI Coding Assistants
Common commercial products in this category: GitHub Copilot, Cursor, Tabnine
- You measure the performance of large language models to solve real-world software problems.
- Using Docker-based infrastructure, you evaluate software development tasks locally or in a cloud environment.
- You analyze the error-solving success of models in different software fields, including multimodal capabilities.
- • NeutralHacker News
Top model scores may be skewed by Git history leaks in SWE-bench
“You apparently don't know what SWE-bench is [1]. First of all, it tries to evaluate skills that explicitly go beyond blank project questions with given solutions. Secondly, it does not contain "optimal" or sometimes even correct solutions. That's because it uses real world…”
- • NeutralHacker News
Some critical issues with the SWE-bench dataset
“The entire premise of this paper is false. They claim that the "hints_text" is used and leaks the answer in Section 2.1.1; however, the authors of SWE-Bench themselves state that this is not used anywhere (Issue #133 on the official SWE-Bench GitHub). According to the paper: >…”
My note
Alternative pages
More in: AI Coding Assistants
- Pi Agent Harness★ 119,051 · today
- Gemini CLI (Google)★ 108,640 · today
- Cline★ 71,082 · today
- open-code-review★ 49,140 · yesterday
- agentic-awesome-skills★ 48,860 · today
- Continue★ 36,380 · yesterday