rsi.watch

Haoyu Zhao et al. (19 authors) · 2026-10-25

Continual Learning Exam: Benchmarking Long-Horizon RSI and Experiential Learning in LLM Agents

“Across 17 harness-model configurations using frontier LLMs, feedback and revision yield a median completion gain of 5.3 percentage points. Yet the best-performing system clears only 47.7% of levels and completes all five levels in just 23.3% of tasks, highlighting the difficulty of sustained learning from experience for existing frontier LLMs.”
Who
Haoyu Zhao et al. (19 authors)
Date
2026-10-25
Evidence
Low certainty Starts at High: measured by someone other than the developer.
  • ↓ indirect measure. The 5.33-point median gain mixes feedback with a second submission. The paper reports no run that resets the harness before each task, so the share of the gain from knowledge retained across tasks is not measured. “Comparing Trials 1 and 2 assesses diagnostic feedback and an additional revision opportunity together.”
  • ↓ imprecise. Each configuration ran once per trial, and the paper reports no spread across agent runs. “The trial labels denote two complete benchmark runs with different per-level submission budgets.”
how we rate evidence
Replication
No rerun found. The paper links a project page but no code or data. We found no rerun by another group. Searched up to 2026-10-28. all claims
Checked
2026-10-28, against the source page
Links
Source Paper

Also on the page

  • CLE has 120 tasks written by 50+ domain experts. Each task is a ladder of five work assignments in one professional role, 600 levels in all, and an agent reaches the next level only after passing the current one. Model weights stay frozen and the persistent harness state carries across all 120 tasks. Trial 1 allows one submission per level. Trial 2 returns rubric-based feedback after a failed submission and allows one more.
  • The authors ran 17 configurations of four harnesses and 11 models. 16 scored higher in Trial 2, and the median gain across all 17 was 5.33 percentage points. The best, Claude Code with Opus 5, cleared 42.17% of levels in Trial 1 and 47.67% in Trial 2, and finished all five levels in 28 of the 120 tasks in Trial 2. Sonnet 5 gained 13.17 points between the trials, against 5.50 for Opus 5.
  • The paper describes RSI as "retained changes that improve how subsequent improvements are generated, evaluated, or implemented". It adds that online learning, test-time training and case-based reasoning "can all operate with fixed learning procedures; their use alone does not establish RSI."
  • The authors say comparing the two trials "assesses diagnostic feedback and an additional revision opportunity together". A run that resets the harness before each task "would assess the contribution of knowledge retained across tasks". Every verdict comes from an evaluation agent on DeepSeek V4 Flash, which agreed with human labels on 96.86% of rubric items in a validation sample.

← Ledger Report an error