Benchmarks / SWE-bench Verified
SWE-bench Verified
What SWE-bench Verified measures, in its makers' own words, and the 2 claims on the rsi.watch ledger that cite it. We do not list scores; its own page has those.
- What it measures
- “We’re releasing a human-validated subset of SWE-bench that more reliably evaluates AI models’ ability to solve real-world software issues.”
- Tasks
- “Together with the authors of SWE-bench, we are releasing SWE-bench Verified: a subset of the original test set from SWE-bench, consisting of 500 samples verified to be non-problematic by our human annotators.”
- How it is scored
- “A proposed edit is evaluated by running both the FAIL_TO_PASS and PASS_TO_PASS tests.”
Limits its makers state
- “Evaluations based on static datasets are inherently limited, and SWE-bench is no exception. Given that the benchmark is composed of scrapes of public GitHub repos, large foundation models that are pre-trained on internet text are likely to be contaminated on the tasks.”
- “As discussed previously, this filtering process is likely to be overzealous but allows us to have high confidence in the feasibility of the unfiltered samples.”
- “The increase in performance when evaluating on SWE-bench Verified may partly be explained by shifting the distribution toward easier samples (as shown in earlier analyses). However, our goal is not to inflate benchmark scores, but to make sure that the benchmark faithfully represents model capability at any given difficulty level.”
Before comparing numbers
- The scoring sentence comes from the page's section on the original SWE-bench. The same section adds: "Both sets of tests are required to pass for the edit to fully resolve the original GitHub issue."
- The Darwin Gödel Machine paper writes SWE-bench for SWE-bench Verified. It says that throughout the paper the term SWE-bench refers by default to the Verified subset.
- DGM scored agents on subsets of Verified: 10 tasks, then 50 more to make 60, then 200 tasks (140 plus the previous 60) for agents above 40% on the 60-task subset that rank in the top two. Its 20.0% to 50.0% is from that staged setup, not from a run on all 500 samples.
On the ledger
- Sakana AI: The Darwin Gödel Machine 2025-05-30On SWE-bench, the DGM automatically improved its performance from 20.0% to 50.0%.Another group (Huxley-Gödel Machine, KAUST) reran DGM as a baseline for its own method, with other models and a 60-task subset of SWE-bench Verified: DGM went from 40% to 53.3% there, and on Polyglot from 20.3% to 27.1%.
- Sakana AI: MASS 2026-10-12Score per output token is 0.96× the base value on Terminal-Bench 2.0 and 0.94× on SWE-bench Verified
OpenAI's page shows only an update date, February 24, 2025, and no first-release date, so no release date is given here.
Other benchmarks the ledger cites Aider Polyglot, KernelBench, RSI-Inference Bench, Terminal-Bench 2.0