rsi.watch

Jiawei Lu et al. (6 authors) · 2026-10-26

RSIV-Harness: A Recursive Self-Improvement Harness Framework for Open-Domain Conversational Text-to-Visualization

“Experiments show that a single round of RSIV-Harness improves all six evaluation metrics relative to the seed harness, with particularly large gains in executability (+28.02 points) and chart structure (+34.01 points)”
Who
Jiawei Lu et al. (6 authors)
Date
2026-10-26
Evidence
Very low certainty Starts at Low: the developer’s own numbers.
  • ↓ risk of bias. The benchmark and the method come from the same paper, and the authors ran every baseline on it themselves. “We introduce ODAC-Vis and RSIV-Harness as a co-designed benchmark and method response.”
  • ↓ inconsistent results. With DeepSeek the evolved harness leads on five of six metrics; with GPT-6 Luna it is slightly behind the strongest baseline on most. The seed harness's DeepSeek executability is 64.98 in the main table and 51.36 in the ablation table. “The transferred system is nevertheless slightly below the strongest GPT baseline on most individual metrics.”
  • ↓ imprecise. One round of evolution for the main result and 96 test tasks, with default temperature. The paper reports no repeated runs, spread or intervals. “We first perform one self-improvement round of RSIV-Harness with DeepSeek-V4.1-Flash.”
how we rate evidence
Replication
No rerun found. The paper links no code. We found no rerun by another group. Searched up to 2026-10-28. all claims
Checked
2026-10-28, against the source page
Links
Source Paper

Also on the page

  • A chart agent answers each dialogue turn under its current harness, with model weights fixed. An outer loop reads failure traces from a panel of 48 training tasks and proposes edits to the prompt, tools, memory, skills and middleware. A revision is kept only if it passes an anchor check on 48 validation tasks and a regression check on stored records; otherwise the best validated harness is restored.
  • The benchmark, ODAC-Vis, is introduced in the same paper: 640 tasks over the 206 Spider databases, where the agent must find the right database, ask clarifying questions and refine charts across turns. Results are on its 96-task test split, whose 15 domains never appear in training or validation. For the main result, one round of evolution was run, with DeepSeek-V4.1-Flash.
  • With DeepSeek, executability goes from 64.98 for the fixed seed harness to 93.00 and chart structure from 41.46 to 75.47. Execution accuracy is 31.52, 0.39 points below direct prompting. The same harness run unchanged with GPT-6 Luna improves five of six metrics over the GPT seed harness, but is "slightly below the strongest GPT baseline on most individual metrics".
  • The ablation table gives the seed harness's executability as 51.36, against 64.98 in the main table. The paper has no limitations section; the authors note that "some performance gains remain dependent on the model used during evolution".

In the self-improving AI catalog

  • RSIV-Harness: what it changes, whether the change lasts, and how it was tested

How it got here

Earlier work in this catalog that RSIV-Harness cites, oldest first. Under each: where that work sits on the map, then where RSIV-Harness sits (changes in bold), and what RSIV-Harness' own paper says about it. Its whole family on the citation map

  1. 2023 Reflexion sets itself apart

    Prompts, memory → Tools, own code · gone when the run ends → saved for later runs · does not improve the improver → improves the task agent

    “Inference-time self-correction methods such as Reflexion, Self-Refine, and DSPy (Shinn et al., 2023; Madaan et al., 2023; Khattab et al., 2023) improve individual trajectories through verbal feedback, iterative drafting, or compiled prompt pipelines, but do not commit persistent harness revisions validated across a task distribution.”

  2. 2023 Self-Refine sets itself apart

    Prompts, memory → Tools, own code · gone when the run ends → saved for later runs · does not improve the improver → improves the task agent

    “Inference-time self-correction methods such as Reflexion, Self-Refine, and DSPy (Shinn et al., 2023; Madaan et al., 2023; Khattab et al., 2023) improve individual trajectories through verbal feedback, iterative drafting, or compiled prompt pipelines, but do not commit persistent harness revisions validated across a task distribution.”

  3. 2026 RSIV-Harness

    Tools, own code · saved for later runs · improves the task agent

← Ledger Report an error