Understanding SWE-bench

A coding system gets a real GitHub issue, a checkout from just before the human fix, and no list of target files. Its answer has to be a patch that survives tests added by the original fix. That combines two different skills: writing code and finding the small change that makes a large repository behave correctly.

A headline such as "76% on SWE-bench" leaves out most of that experiment. The score belongs to a system: benchmark variant and revision, model, agent scaffold, tools, attempt rule, budget, and harness. Change one of those and the percentage answers a different question.

A patch inside a real repository

HumanEval, a 164-function Python set, gives a model a function signature and docstring. MBPP (Mostly Basic Python Problems) gives a short programming prompt and tests. Both mainly test small, isolated program synthesis: the coding target is already named.^{[1]Reference 1Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374 [2]Reference 2Program Synthesis with Large Language Modelshttps://arxiv.org/abs/2108.07732}

SWE-bench starts one step earlier. It pairs a GitHub issue with the repository state from before the human fix, and asks for a patch rather than an explanation.^{[3]Reference 3SWE-bench: Can Language Models Resolve Real-World GitHub Issues?.https://arxiv.org/abs/2310.06770} The system has to decide where to look before it can decide what to change.

The original test split contains 2,294 instances from 12 Python repositories. A gold fix changes 1.7 files and 32.8 lines on average, while each repository contains about 438,000 non-test lines.^{[3]Reference 3SWE-bench: Can Language Models Resolve Real-World GitHub Issues?.https://arxiv.org/abs/2310.06770} Most of that tree is irrelevant to any one issue. Localization is part of the benchmark, not preparation outside it.

How a task is constructed

Each instance starts with a resolved issue and its merged pull request (PR). The dataset pins the repository's pre-fix base_commit, then splits the PR into a gold patch (the accepted non-test change) and a test patch (tests added or changed to expose the repaired behavior).^{[3]Reference 3SWE-bench: Can Language Models Resolve Real-World GitHub Issues?.https://arxiv.org/abs/2310.06770}

The builder runs those tests against both sides of the history pair. Tests that flip from failing to passing become FAIL_TO_PASS; tests that were already green and remain green become PASS_TO_PASS. Those two hidden-test lists turn the issue into a repair gate: a patch can't "succeed" by fixing one path while breaking covered behavior.

The candidate therefore sees the issue and checkout, but not the evaluator-only patches, selected test lists, or optional hints. Existing tests can provide local feedback, but the system doesn't know which cases the harness will use as its final gates.^{[4]Reference 4SWE-bench Source Repositoryhttps://github.com/SWE-bench/SWE-bench [5]Reference 5SWE-bench Experimentshttps://github.com/SWE-bench/experiments}

The gold patch is evidence of one accepted solution, not a required edit script. A different patch should pass when it implements the requested behavior. That promise fails when a hidden test checks an unmentioned helper name or another detail tied to the reference implementation. The 2026 audits found repeated examples of this verifier problem.^{[6]Reference 6Why SWE-bench Verified no longer measures frontier coding capabilitieshttps://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/ [7]Reference 7Separating signal from noise in coding evaluationshttps://openai.com/index/separating-signal-from-noise-coding-evaluations/}

What a coding system must do

SWE-bench doesn't require one agent architecture. A non-interactive pipeline can retrieve files and generate a patch; an interactive agent can search, edit, run tests, and retry. Agentless showed that localization, candidate generation, and reranking can be competitive without a long autonomous loop.^{[8]Reference 8Agentless: Demystifying LLM-based Software Engineering Agentshttps://arxiv.org/abs/2407.01489}

For an interactive system, the useful question is what happens between issue and submission. A compact repair cycle gives each tool call a job:

Consider this fictional issue:

LogisticRegression ignores max_iter when solver="liblinear".

A disciplined trace starts by searching for LogisticRegression, liblinear, and nearby tests. It reproduces the ignored value, follows the solver branch, makes the smallest argument handoff, and reruns the reproduction before broader focused tests. This isn't a SWE-bench instance. It's a small trace of the evidence a repository repair requires.

Tool design controls how much useful evidence that trace produces. SWE-agent calls its tool layer an agent-computer interface (ACI): commands for search, file viewing, editing, execution, and concise feedback.^{[9]Reference 9SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringhttps://arxiv.org/abs/2405.15793} The AI Coding Workflow with Agents lesson turns those mechanics into a bounded engineering workflow with file ownership, before-and-after evidence, and human merge authority. The Code Generation & Sandboxing lesson supplies the runtime boundary: the model proposes a patch, a trusted host admits the job, and tests run in an isolated environment.

How the harness grades a patch

Once a candidate returns a patch, the harness follows a fixed path. It starts the task-specific container, applies the candidate patch, applies the evaluator's test patch, runs the task's evaluation script, parses its output, and checks both required test sets.^{[4]Reference 4SWE-bench Source Repositoryhttps://github.com/SWE-bench/SWE-bench} The gold patch stays out of this path; it helped construct the task but doesn't rescue the candidate.

Take two task rows: one patch applies and clears both gates, while another misses one regression test. The first row contributes one resolved task; the second contributes zero. For task $i$, write that verdict as $r_i=1$ only when the patch applies and every required test passes; otherwise $r_i=0$. Across $N$ declared tasks:

$$\text{Resolved rate} = \frac{1}{N}\sum_{i=1}^{N} r_i$$

Use the full declared split as $N$ when reading an official row. A missing prediction, patch-application error, timeout, or infrastructure failure is a failed row unless the published protocol explicitly excludes it. Quietly removing hard rows makes the numerator look better without improving the system.

The harness can record how many FAIL_TO_PASS tests passed, but that diagnostic isn't public credit. A patch that clears some fix tests and all regressions still scores zero for the task.^{[10]Reference 10SWE-bench Official Leaderboardshttps://www.swebench.com/}

A candidate passes every FAIL_TO_PASS test but fails one PASS_TO_PASS regression test. What resolved rate should the task receive?

Answer

Zero. SWE-bench's task-level result requires all required tests, including regressions, to pass. Treat the failure as evidence about the patch or harness instead of counting partial progress as a resolved task.

That all-or-nothing gate rejects plausible-looking partial patches. It doesn't guarantee a fair grade: tests can be too narrow, too broad, underspecified, or coupled to one implementation. The score is only as meaningful as both the task and its verifier.

Choose the benchmark before reading a score

The label "SWE-bench" now covers several official test sets plus a related successor. Their percentages aren't interchangeable because each label changes the task population. The table is a dated snapshot checked August 13, 2026 against the official leaderboard and primary dataset sources.^{[10]Reference 10SWE-bench Official Leaderboardshttps://www.swebench.com/ [3]Reference 3SWE-bench: Can Language Models Resolve Real-World GitHub Issues?.https://arxiv.org/abs/2310.06770 [11]Reference 11SWE-bench Lite: A Filtered Subset for Practical Evaluationhttps://www.swebench.com/lite.html [12]Reference 12Introducing SWE-bench Verifiedhttps://openai.com/index/introducing-swe-bench-verified/ [13]Reference 13SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?https://arxiv.org/abs/2410.03859 [14]Reference 14SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?https://arxiv.org/abs/2509.16941}

Lite is cheaper for iteration because its selection removes several expensive or ambiguous shapes. It drops instances with images, external links, commit SHAs, pointers to other issues or PRs, fewer than 40 words of issue text, gold patches that edit more than one file or more than three hunks, file create/delete, and tests that check error-message strings.^{[11]Reference 11SWE-bench Lite: A Filtered Subset for Practical Evaluationhttps://www.swebench.com/lite.html} Multi-file and visually grounded repair therefore disappear from this subset.

The project repository's current dataset guide has a conflicting Lite count, while the official Lite page and leaderboard define the public test set as 300 tasks. For historical comparisons, use that original 300-task definition; for a new run, pin the dataset revision and count the split you actually loaded.^{[11]Reference 11SWE-bench Lite: A Filtered Subset for Practical Evaluationhttps://www.swebench.com/lite.html[4]Reference 4SWE-bench Source Repositoryhttps://github.com/SWE-bench/SWE-bench}

Verified is a 500-instance subset created after professional developers reviewed 1,699 original problems, with three experts independently reviewing each problem during construction.^{[12]Reference 12Introducing SWE-bench Verifiedhttps://openai.com/index/introducing-swe-bench-verified/ [6]Reference 6Why SWE-bench Verified no longer measures frontier coding capabilitieshttps://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/} That curation improved task quality, but the set remains public and heavily studied.

Multilingual changes both language and repository mix. Multimodal's paper describes a 619-instance collection across 17 JavaScript repositories; its scored public test split has 517 tasks from 12 mostly JavaScript repositories, and every task has an image in the issue or tests.^{[13]Reference 13SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?https://arxiv.org/abs/2410.03859 [10]Reference 10SWE-bench Official Leaderboardshttps://www.swebench.com/}

SWE-Bench Pro is a separate Scale AI benchmark, not another tab on the official SWE-bench leaderboard. Its 1,865 human-verified tasks come from 41 repositories: 731 public, 858 held out, and 276 commercial.^{[14]Reference 14SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?https://arxiv.org/abs/2509.16941} The public set uses copyleft-licensed repositories by design. None of these rows is a drop-in denominator for another.

What a leaderboard row records

An unrestricted leaderboard row is a system result. It mixes the model with its scaffold, tools, prompt, budget, and retry policy, so it can compare usable systems but can't isolate model quality.

The official Verified leaderboard offers a Bash Only view to narrow that variation. Every model runs inside the same mini-SWE-agent environment with a bash shell and a simple ReAct loop, without an extra product scaffold.^{[10]Reference 10SWE-bench Official Leaderboardshttps://www.swebench.com/} The release number identifies the mini-SWE-agent version. Versions 1.x and 2.x aren't directly comparable: 1.x parsed actions from model output text, while 2.x invokes actions with tool calling. The recorded temperature also differs: 1.x uses 0.0 when supported, while 2.x leaves it unset.

Comparison rule: Compare model rows only when variant, dataset revision, scaffold version, tools, attempt policy, and budgets match. Otherwise describe a system comparison.

This dated snapshot puts protocol next to score. All three rows used Verified, mini-SWE-agent 2.0.0, high reasoning effort, and one submitted patch per task. The submissions are dated February 17, 2026; the live leaderboard was checked on August 13, 2026.^{[10]Reference 10SWE-bench Official Leaderboardshttps://www.swebench.com/}

Read that table narrowly. Under this scaffold and dataset, the resolved rates were close while recorded inference cost differed by about 10x. It doesn't establish equal product quality, current API pricing, or performance under another agent.

The submission policy also separates attempt rules. Treat these as different claims, even when they use the same task set:^{[5]Reference 5SWE-bench Experimentshttps://github.com/SWE-bench/experiments}

pass@k answers a search question: did any of several sampled patches work? It isn't the single-submission SWE-bench headline. Evaluating AI Agents derives pass@k, repeatability metrics, hard gates, and cost per successful task.

Before comparing a row, ask:

• Which variant and exact dataset revision?
• Which model snapshot and agent or retrieval scaffold?
• Which tools, system prompt, image access, and test access?
• How many attempts, steps, tokens, and retries per task?
• What selection rule turned multiple patches into one submission?
• What were total cost, latency, setup failures, and timeouts?
• Are predictions, trajectories, logs, and harness reports available?
• If the system can browse the web, what blocked it from fetching the original PR?

Leaderboard coverage also reflects eligibility. Since November 18, 2025, new Verified and Multilingual entries require an academic or research affiliation plus a public arXiv preprint or technical report. Earlier submissions remain, and Multimodal remains open to other submitters.^{[5]Reference 5SWE-bench Experimentshttps://github.com/SWE-bench/experiments} Absence from a table isn't evidence that a product failed the benchmark.

What the 2026 audits changed

Reproducibility doesn't make a benchmark valid by itself. SWE-bench Verified remains an official public test set, but OpenAI stopped reporting it for frontier launches on February 23, 2026. Its announcement named two separate concerns: flawed tasks in a targeted audit and training exposure to public issues and solutions.^{[6]Reference 6Why SWE-bench Verified no longer measures frontier coding capabilitieshttps://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/}

The denominator matters. Reviewers examined 138 of 500 tasks that OpenAI o3 did not solve consistently across 64 runs, a 27.6% slice. They found material test-design or problem-description issues in 59.4% of that selected subset. OpenAI's breakdown of the audited slice was 35.5% overly narrow tests, 18.8% tests that check extra unspecified behavior, and 5.1% miscellaneous issues. The result isn't an estimate that 59.4% of all Verified tasks are broken.

Here is the verifier problem in concrete form. In pylint-dev__pylint-4551, the issue asks pyreverse to read Python type hints, while hidden tests import a new helper named get_annotation that the issue never mentions. A patch can implement the requested behavior and still fail with ImportError.^{[6]Reference 6Why SWE-bench Verified no longer measures frontier coding capabilitieshttps://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/}

The same audit found that every frontier model tested could reproduce a gold patch or a verbatim task-specific detail for at least some cases. That's evidence of contamination somewhere in the set, not proof that every successful patch was memorized. Exposure matters most when an underspecified prompt leaves details that only the hidden test or original solution reveals.

SWE-Bench Pro already existed when OpenAI published the Verified audit. OpenAI recommended Pro in February, then retracted that recommendation on July 8, 2026 after auditing Pro's 731-task public split.^{[14]Reference 14SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?https://arxiv.org/abs/2509.16941 [7]Reference 7Separating signal from noise in coding evaluationshttps://openai.com/index/separating-signal-from-noise-coding-evaluations/} An agent-assisted pipeline flagged 200 tasks (27.4%); a human annotation campaign identified 249 (34.1%). OpenAI summarized the result as roughly 30%, citing four patterns:

• tests that require unmentioned implementation details
• prompts that omit behavior enforced by tests
• tests too weak to catch incomplete solutions
• prompts that point toward behavior inconsistent with tests

These audits don't make every public result useless. They narrow the claim. Verified remains useful for historical and reproducible system comparisons, especially when the scaffold is open, but public exposure makes it weak evidence for fine-grained frontier ranking. Pro may still diagnose systems on its surviving tasks; an aggregate score across a disputed public split needs task-level quality analysis. LLM Benchmarks & Limitations carries the broader hygiene lesson: name the split, harness, and failure mode before treating a percentage as capability.

Evidence rule: Treat a public benchmark score as screening evidence. Use task-level traces and an internal held-out suite before making a product or release decision.

Reproduce a claim before comparing it

To compare a claim, start with one leaderboard row or paper table. Save a run manifest that preserves the protocol rather than relying on a screenshot or memory:

Keep each patch, observable trace, test log, harness report, cost, and latency beside that manifest. Recompute the resolved rate over the declared task IDs, and label missing predictions or infrastructure failures instead of hiding them.

Run a small fixed slice before spending on the full set. Check checkout, dependency setup, patch application, test discovery, timeout behavior, and report parsing. That smoke test validates plumbing, not the benchmark score. If dataset containers or provider endpoints have drifted, record the mismatch instead of comparing unlike runs.

For a model comparison, hold scaffold, task IDs, tools, image, prompt, attempt policy, and budgets constant. If any of those change, call it a system comparison.

What SWE-bench leaves untested

A high score is evidence of targeted repository repair under one harness. It doesn't answer whether an agent can own:

• greenfield architecture and product discovery
• multi-week migrations and coordination
• code review, security review, and performance review
• deployment, rollback, and incident ownership
• private repository conventions and weak local test suites
• your languages, frameworks, permissions, and latency budget

Build a private suite from representative repository work that wasn't used to tune the agent. Include narrow fixes, cross-file changes, ambiguous requests that should trigger clarification, unsafe instructions that should be rejected, flaky setup, and tasks where the correct response is to stop. Grade final behavior, regressions, scope, cost, latency, and review burden alongside test pass rate.

Practice: trace the verdict

Use the same verdict rule on a tiny fictional issue:

mean([]) must raise StatisticsError, but the current implementation returns 0.0.

The harness evaluates candidate patches against both fix requirements (FAIL_TO_PASS) and regression protection (PASS_TO_PASS):

Three rows tell three different failure stories. The buggy patch preserves the issue, so it fails FAIL_TO_PASS. A regressive patch raises the requested exception but breaks non-empty inputs, so it fails PASS_TO_PASS. Only the third patch clears both gates and counts as resolved.

Two habits transfer to real agent work. Review a patch against exact behavior, not whether the diff looks plausible. Preserve the run context that explains every verdict. Build that workflow in AI Coding Workflow with Agents, then use Evaluating AI Agents to design private episodes, repeatability checks, and release gates around it.