rsi.watch

Apple ML Research · 2026-10

RSI-Inference Bench: A Long-Horizon Benchmark for Self-Improving Inference Systems

“RSI-Inference Bench is a long-horizon benchmark for coding agents that improve the inference stack of large language models.”
Who
Apple ML Research
Date
2026-10 (month only) The README and the leaderboard give no release date. The leaderboard data are dated 2026-10-25.
Evidence
Low certainty Starts at High: measured by someone other than the developer.
  • ↓ indirect measure. The rating is of the leaderboard results in the data. The agents edit vLLM or SGLang to serve other models. A next round, in which an improved engine serves the agent's own model, is not part of the benchmark. “RSI-Inference Bench is a long-horizon benchmark for coding agents that improve the inference stack of large language models.”
  • ↓ imprecise. Day-0 Model Support has 10 tasks, most with 3 attempts per agent. Gemini-4-Argon's pass@1 of .0667 rests on 2 full passes on one task. The data give low and high values for pass@1 and speed-up without saying how they are computed, and none for the joint rating. “Full passes over attempts on each task, averaged over the tasks.”
how we rate evidence
Replication
No rerun found. The tasks and the runner are in the repository. We found no rerun by another group. Searched up to 2026-10-28. all claims
Checked
2026-10-28, against the source page
Links
Source Code

Also on the page

  • The benchmark has 37 tasks in three stages. Day-0 Model Support (10 tasks) is scored pass or fail and reported as pass@1. Inference Optimization (20 tasks) and Bug Fix (7 tasks) are scored by speed-up, reported as a geometric mean over the 27 tasks. An agent works offline for up to 8 hours on 1 to 8 GPUs.
  • In the leaderboard data of 2026-10-25, Sonnet-5.5 ranks first of 15 agents overall and first in both tracks, with pass@1 .2667 on Day-0 Model Support and a speed-up of 1.6772 on the 27-task speed-up track. Opus-5.5 is second in both, at .1333 and 1.4839.
  • Gemini-4-Argon ranks 6th overall, with a joint rating of 1675.64 against 1676.18 for GPT-6-Astra in 5th. On Day-0 Model Support it has pass@1 .0667, from 2 full passes in 30 attempts, both on the GLM-5.3-Flash task, and passed a share of .8098 of the hidden tests, averaged over tasks. On the 27-task speed-up track it ranks 4th, with 1.3937.
  • The authors are Junteng Liu, Richard He Bai, Huangjie Zheng, Yizhe Zhang and Ruixiang Zhang. The README marks the paper as coming soon.

Benchmarks it cites

RSI-Inference Bench: what each measures, in its makers' words.

← Ledger Report an error