rsi.watch

Hau-Sing So et al. (3 authors) · 2026-10-28

Masked Erosion: How Acceptance Gates Lose Redundant Safeguards in Self-Improving Agents

“Large harm audits prevent exposure but not the loss of depth, which turns into harm when the harness is reused with a weaker model.”
Who
Hau-Sing So et al. (3 authors)
Date
2026-10-28
Evidence
Very low certainty Starts at Low: the developer’s own numbers.
  • ↓ indirect measure. 393 of the 411 loops plant a deletion each generation rather than let an optimizer propose edits. The harm after reuse with another model comes mainly from Qwen models that solve about 5% of clean AppWorld tasks. The authors check whether that is a weak model trying things at random: under Qwen3-30B, finals that kept auth and at least one other layer misused credentials in 2 of 200 runs, auth alone in 9 of 40, and finals without auth in 89 of 200. Among protected finals the gap was 0.025 under Luna and Sol, and 0.020 after a task shift under DeepSeek (p = .061). “We study two environments, each with one harm and four written safeguards, and most loops plant one deletion per generation to isolate the gate.”
  • ↓ imprecise. Most gate comparisons rest on six loops per arm, the model swap on 11 distinct finals at 40 runs each, and GEPA on three seeds. The Qwen3-30B gap among protected finals has a 95% interval of [0.16, 0.47]; for GEPA's seed 2, a sign test over tasks gives p = .125. “With six loops per arm, Section 4.3 reports exact loop-level permutation tests.”
how we rate evidence
Replication
No rerun found. The paper says "We will release code, the exact safeguard texts and proposer prompts, frozen predictions with hashes and run records" and links no repository. We found no rerun by another group. Searched up to 2026-10-30. all claims
Checked
2026-10-30, against the source page
Links
Source Paper

Also on the page

  • A study of the acceptance gate that self-improving agents use to keep or reject an edit to their own harness; in the authors' words, "We study the acceptance step these systems share", and the paper proposes no new self-improving system. The agent's harness carries four written safeguards in two environments: credential misuse in AppWorld and destructive clean-up in OpsSim, a simulated operations service. Of the 411 loops, 393 are stress loops that propose deleting one uniformly chosen safeguard per generation, and 18 are natural loops in which a DeepSeek proposer rewrites the harness to make it cheaper. Three more runs use GEPA unchanged.
  • In 27 AppWorld stress loops with DeepSeek-V4.1-Flash, greedy loops ended unprotected in 5 of 6 runs and loops under the paired gate, a one-sided exact McNemar test, in none. The authors add that the paired test "accepted 2 of 42 masked deletions", that chaining its rates gives an unprotected final with "probability 0.19 by generation 64", and that its protection comes from conservatism: at alpha .05 it passes a real +0.075 edit in 8% of decisions, against 96% for greedy. The 32 deletions into still-protected harnesses that the greedy, split-half and paired gates accepted came with 114 harm-pool tasks, 48 of them distinct, and no child-only harm.
  • Where a safeguard costs task success, the paired gate deletes it too. In three-step OpsSim it accepted deleting confirm, the only sufficient layer for DeepSeek, Luna and Sol, in 15 of 20 proposals from the full stack. In four-step OpsSim, where the layers are free and the authors gave confirm a cost by letting the owner it asks stay silent on some tasks, at a cost of 0.15 paired loops with 20-task batches kept all six harnesses protected, where 40-task loops lost all six.
  • Comparing only finals still protected under DeepSeek, non-paired finals misused credentials in 0.30 of runs under Qwen3-30B and paired finals in 0.004 (CI of the difference [0.16, 0.47]); under Qwen3-8B the gap was 0.16. Both Qwen models solve about 5% of clean AppWorld tasks. Under GPT-6 Luna and GPT-6.1 Sol the gap was 0.025, and the authors write: "The effect is specific to weaker executors."
  • GEPA, scored on task success alone, kept the credential rules verbatim in all three finals, and its finals solved 0.35 to 0.45 of clean Tfinal tasks against 0.40 for the starting harness. Two finals rewrote the note template into an instruction to log in with the stored credentials; under Qwen3-30B they misused credentials in 2 and 5 of 40 runs, against 0 of 80 for the starting harness. The authors: "With three seeds and few harm events, it is a case study, not an estimate."
  • Freezing only confirm kept all six three-step OpsSim loops protected, while the costly layers around it were deleted, at a cost of 0.05 in clean success; a margin-aware gate, which accepts a child only if every one-layer ablation stays protected, kept every loop protected there at a cost of 0.35. In all three natural loops the margin-aware gate let verify be weakened.
  • Of 177 registered predictions, 113 held, 49 failed, 9 held in part and 6 could not be scored. Of the 14 tests behind the main claims, 7 held; 3 fell short in the predicted direction, 2 found a stronger effect than predicted and 2 found a harm-aware gate weaker than predicted. The paper says "Three later prediction sets, including GEPA’s, were frozen after related outcomes, and the drift account was formulated after a predicted AppWorld selection effect failed to replicate."
  • The authors' limits: "We chose this setting to expose how gates respond to deletion pressure, not to estimate how often deployed optimizers erode safeguards; the natural loops and GEPA runs are case studies." Also: "results from the third-party API models are supplementary."

← Ledger Report an error