Benchmarks / RSI-Inference Bench
RSI-Inference Bench
What RSI-Inference Bench measures, in its makers' own words, and the claim on the rsi.watch ledger that cites it. We do not list scores; its own page has those.
- What it measures
- “a long-horizon benchmark for coding agents that improve the inference stack of large language models”
- Tasks
- “It follows a model through its life in a production engine, vLLM or SGLang: supported on day 0, optimized after launch, and repaired when a regression lands.”
- How it is scored
- “a separate verifier builds the edited engine, serves the model and measures it; the score is pass or fail, or the measured speed-up.”
Before comparing numbers
- There are 37 tasks: 10 for Day-0 Model Support, 20 for Inference Optimization and 7 for Bug Fix. The leaderboard puts the last two together in one track of 27 tasks.
- Day-0 Model Support is reported as pass@1, and a partial pass counts as a failure. Ties are broken by the share of hidden tests passed, which the leaderboard data call f2p.
- The speed-up track is a geometric mean of each task's ratio to its baseline. A result that fails its correctness check counts as 1.00×.
- The overall order comes from a joint rating over both tracks, on the Elo scale. The pages we saved state no limits, and the paper is marked as coming soon.
On the ledger
- Apple ML Research: RSI-Inference Bench 2026-10In the leaderboard data of 2026-10-25, Sonnet-5.5 ranks first of 15 agents overall and first in both tracks, with pass@1 .2667 on Day-0 Model Support and a speed-up of 1.6772 on the 27-task speed-up track. Opus-5.5 is second in both, at .1333 and 1.4839.Gemini-4-Argon ranks 6th overall, with a joint rating of 1675.64 against 1676.18 for GPT-6-Astra in 5th.
The README and the leaderboard give no release date. The leaderboard data are dated 2026-10-25.
Other benchmarks the ledger cites Aider Polyglot, KernelBench, SWE-bench Verified, Terminal-Bench 2.0