Benchmarks / Aider Polyglot
Aider Polyglot
What Aider Polyglot measures, in its makers' own words, and the claim on the rsi.watch ledger that cites it. We do not list scores; its own page has those.
- What it measures
- “more clearly distinguishes the performance of today’s strongest coding models and leaves headroom for future LLMs.”
- Tasks
- “The new benchmark uses the 225 problems that were solved by 3 or fewer models.”
Before comparing numbers
- The problems are in C++, Go, Java, JavaScript, Python and Rust. Aider's post gives no scoring rule; its results table reports percent completed correctly.
- DGM scored Polyglot as pass@1, where the agent never sees ground-truth test results. DGM's paper says Aider's leaderboard reports pass@2, where the agent can view feedback from ground-truth tests once. DGM's numbers and Aider leaderboard numbers come from different settings.
- DGM's own run used a 50-task subset (14.0% to 38.0%). The 14.2% to 30.7% pair comes from evaluating the base agent and the best DGM agent on the full Polyglot benchmark.
- The Huxley-Gödel Machine paper says it follows DGM's evaluation setting, in which agents get no access to test results.
On the ledger
- Sakana AI: The Darwin Gödel Machine 2025-05-30On Polyglot it went from 14.2% to 30.7%.Another group (Huxley-Gödel Machine, KAUST) reran DGM as a baseline for its own method, with other models and a 60-task subset of SWE-bench Verified: DGM went from 40% to 53.3% there, and on Polyglot from 20.3% to 27.1%.
Date line on Aider's post introducing the benchmark.
Other benchmarks the ledger cites KernelBench, RSI-Inference Bench, SWE-bench Verified, Terminal-Bench 2.0