rsi.watch

Hao Chen et al. (10 authors) · 2026-10-27

When the Critic Falls Behind: Co-Evolving Rubrics and Harnesses for Continual Agent Self-Improvement

“Extensive experiments across diverse agent tasks and backbone models show that RuHarness outperforms strong harness-evolution baselines, particularly on unseen held-out tasks.”
Who
Hao Chen et al. (10 authors)
Date
2026-10-27
Evidence
Very low certainty Starts at Low: the developer’s own numbers.
  • ↓ imprecise. The paper gives no intervals for the main table. Repeated runs exist for two settings, three seeds each; on Knowledge Work held-out, RuHarness and AHE score 50.0 and 47.3, with standard deviations of 2.0 and 3.1. Held-out sets have 36 to 100 tasks per domain, and the average gains over the strongest baseline are 4.5 and 5.3 points. “we repeat all methods with three independent random seeds on two representative settings”
how we rate evidence
Replication
No rerun found. The paper links no code. We found no rerun by another group. Searched up to 2026-10-29. all claims
Checked
2026-10-29, against the source page
Links
Source Paper

Also on the page

  • Each round, failed runs are turned into behavioral criteria, an outcome-blind judge scores them on new trajectories, and the top 10 active criteria guide edits to the harness in six coordinates: Equip, Instruct, Act, Observe, Remember and Terminate. The backbone, Qwen3.5-35B-A3B or GLM-4.7-Flash, stays fixed and also runs the evolution. A candidate is kept only if its score on a separate validation split beats the current harness by more than a threshold, set to 0.05.
  • On EvoAgentBench's five domains, average held-out Pass@1 with Qwen3.5-35B-A3B goes from 44.2 for the vanilla agent to 53.7, where the best baseline, AHE, scores 49.2. With GLM-4.7-Flash it goes from 24.5 to 38.0, where Self-Harness scores 32.7. The authors ran the baselines with "Evolution rounds, candidate budgets, and validation-based selection protocols" matched across harness-evolution methods "unless otherwise specified".
  • In the ablation on Qwen3.5-35B-A3B, keeping the initial rubric fixed gives an average held-out score of 44.4, below the 45.8 of the variant with no rubric module.
  • The authors' limits: "all experiments are conducted within a single underlying agent framework", and the experiments run "under a fixed evolution horizon".

In the self-improving AI catalog

  • RuHarness: what it changes, whether the change lasts, and how it was tested

How it got here

Earlier work in this catalog that RuHarness cites, oldest first. Under each: where that work sits on the map, then where RuHarness sits (changes in bold), and what RuHarness' own paper says about it. Its whole family on the citation map

  1. 2026 RRSI sets itself apart

    Tools, own code · saved for later runs → becomes the next version · improves the task agent → improves the improver

    “More broadly, recent studies have explored recursive self-improvement and harness–model co-evolution across repeated adaptation rounds(Xia et al., 2026; Yu et al., 2026; Chen et al., 2026b; Fan and Huang, 2026).”
    “Yet these approaches still largely treat the diagnostic signals guiding evolution as fixed, leaving how such guidance should adapt to shifting behavioral weaknesses underexplored.”

  2. 2026 RuHarness

    Tools, own code · becomes the next version · improves the improver

Also cites Reflexion 2023, Weco AIDE² 2026, Recursive Harness Self-Improvement 2026

← Ledger Report an error