rsi.watch

Wen Jiang et al. (9 authors) · 2026-09-01

HarnessEvolve: Learning from Reference Trajectories for Reliable Agent Self-Evolution

“Notably, on CloudCoreNetwork-QA, HarnessEvolve outperforms the strongest baseline by 21.6 percentage points, demonstrating substantial gains on complex multi-skill enterprise tasks.”
Who
Wen Jiang et al. (9 authors)
Date
2026-09-01
Evidence
Very low certainty Starts at Low: the developer’s own numbers.
  • ↓ imprecise. Each method and setting is reported as one accuracy figure, with no intervals or repeated runs, and the paper does not give the sizes of the training, validation and test splits. “Accuracy (%) is reported. Best results are in bold.”
  • ↓ indirect measure. The 21.6-point lead in the quote is on an in-house dataset others cannot run, with Qwen3.6-27B "post-trained with domain-specific fine-tuning". On the three public benchmarks the leads over the best baseline are smaller: 92.9 against 90.0, 70.9 against 68.9 and 76.4 against 74.6. “CloudCoreNetwork-QA and Wireless-QA, two in-house datasets from cloud core network and wireless network domains”
how we rate evidence
Replication
No rerun found. The paper links no code, and two of the five datasets are in-house. We found no rerun by another group. Searched up to 2026-10-29. all claims
Checked
2026-10-29, against the source page
Links
Source Paper

Also on the page

  • The paper lists the affiliation "ICT AI Competence Center, Huawei Technologies, Shanghai, China". Four agents work in a loop: an execution agent runs tasks with its harness (prompts, skills, tools and execution logic) on a frozen LLM; an evaluation agent marks failures and checks reference trajectories, which the execution agent produces when given the ground-truth answer; an optimization agent compares failed runs with those references, clusters the errors and proposes harness edits; a gate agent checks edits for leaked training answers and too many in-context examples, and a rejected edit goes back to the optimization agent for revision, up to 3 times, then is abandoned.
  • A candidate is accepted if it improves accuracy on the current batch by at least a margin, set to 0.0, and does not fall by more than 0.025 on the 2 previous batches. At each epoch end the best accepted snapshot on the validation set is kept, and the final agent is scored on a separate test split.
  • On CloudCoreNetwork-QA with Qwen3.6-27B, accuracy goes from 43.4% for the unoptimized harness to 86.9%, against 65.3% for GEPA. CloudCoreNetwork-QA and Wireless-QA are in-house datasets, and Qwen3.6-27B is "post-trained with domain-specific fine-tuning". On SearchQA, OfficeQA and SpreadsheetBench with DeepSeek-V4-Flash under OpenClaw, accuracy goes from 86.5 to 92.9, 62.8 to 70.9 and 44.3 to 76.4.
  • In the ablation on CloudCoreNetwork-QA, removing the reference trajectories drops accuracy from 86.9% to 57.8%. Skills optimized on OpenClaw, moved to four other frameworks without re-optimization, "consistently improve or maintain accuracy over Base".

In the self-improving AI catalog

  • HarnessEvolve: what it changes, whether the change lasts, and how it was tested

How it got here

Earlier work in this catalog that HarnessEvolve cites, oldest first. Under each: where that work sits on the map, then where HarnessEvolve sits (changes in bold), and what HarnessEvolve's own paper says about it. Its whole family on the citation map

  1. 2023 Reflexion sets itself apart

    Prompts, memory → Tools, own code · gone when the run ends → becomes the next version · does not improve the improver → improves the task agent

    “Current approaches optimize along multiple dimensions: workflow-level architecture design (Zhang et al., 2025; Liu et al., 2025), prompt-level optimization (e.g., OPRO (Yang et al., 2023), APO (Pryzant et al., 2023), and GEPA (Agrawal et al., 2025)), context-level evolution such as Reflexion (Shinn et al., 2023), ExpeL (Zhao et al., 2024), and ACE (Zhang et al., 2026b), and tool/skill-level refinement including Voyager (Wang et al., 2024) and SkillOpt (Yang et al., 2026).”
    “However, each optimizes only a single component of the harness independently, leaving the rest untouched and limiting the space of achievable improvements.”

  2. 2023 Voyager sets itself apart

    Tools, own code · saved for later runs → becomes the next version · improves the task agent

    “Current approaches optimize along multiple dimensions: workflow-level architecture design (Zhang et al., 2025; Liu et al., 2025), prompt-level optimization (e.g., OPRO (Yang et al., 2023), APO (Pryzant et al., 2023), and GEPA (Agrawal et al., 2025)), context-level evolution such as Reflexion (Shinn et al., 2023), ExpeL (Zhao et al., 2024), and ACE (Zhang et al., 2026b), and tool/skill-level refinement including Voyager (Wang et al., 2024) and SkillOpt (Yang et al., 2026).”
    “However, each optimizes only a single component of the harness independently, leaving the rest untouched and limiting the space of achievable improvements.”

  3. 2026 HarnessEvolve

    Tools, own code · becomes the next version · improves the task agent

← Ledger Report an error