rsi.watch

CRUX · 2026-10-20

Can frontier AI agents conduct open-ended AI research? Follow-up experiments with GPT-6 Astra and Claude Fable 5.1

“In a series of follow-up experiments, we observed meaningful improvements in experimental design and resource management, but persistent limitations in creative problem solving, judgment, calibration, and communication; both papers were rejected by our experts.”
Who
CRUX
Date
2026-10-20 Date of the CRUX research note. CRUX published the first round in July.
Evidence
Low certainty Starts at High: measured by someone other than the developer.
  • ↓ risk of bias. The graders are the authors of the original papers. CRUX raised this caveat about non-blind reviewing in the first-round paper; the follow-up note does not repeat it. For the first round CRUX released the expert reviews so others can check them. “Non-blind reviewing might come with reviewer biases, since reviewers have already answered the question with a specific method and also know that the paper was written by AI agents.”
  • ↓ imprecise. One run per model, each on one research question, so two papers in this round. “Over the last month, we’ve conducted a second round of experiments, one with OpenAI’s GPT-6 Astra and one with Anthropic’s Claude Fable 5.1.”
how we rate evidence
Replication
Rests on internal data. The research questions come from papers that are not yet public. CRUX says it will share the logs and the agents' papers once they are. Searched up to 2026-10-22. all claims
Checked
2026-10-22, against the source page
Links
Source

Also on the page

  • Each agent got the research question of an unpublished paper, 7 days and $10,000 in API credits: GPT-6 Astra in Codex, Fable 5.1 in Claude Code. The original authors graded the papers on a conference reviewer rubric: "Strong Reject" for Fable 5.1 and "Reject" for GPT-6 Astra.
  • Both agents rated their own work higher than the experts did: each gave at least one of its drafts a "Borderline Accept".
  • Fable 5.1 stopped with ~3.7 days and ~$7,500 in API spend remaining. CRUX calls this "a clear violation of an explicit instruction in the prompt".
  • In the first round (Claude Opus 4.8 in OpenClaw, with a check using GPT-5.6 Sol in Codex), "The agents completed all of the engineering without human help, yet could not make substantial progress towards answering the research questions."
  • On the gains reported by RSI benchmarks, CRUX writes "we think the two are consistent": "Most tasks on these benchmarks, such as lossless text compression or improving how a small language model is trained, are much less open-ended than the research questions we test."
  • The first limitation CRUX lists is resources: $10,000 "is still an order of magnitude lower than single experiments run in the labs", and the agents had a week. On time, CRUX writes: "We remain confident that time was not the binding constraint".
  • CRUX thanks OpenAI "for providing API credits to evaluate GPT-6 Astra".

← Ledger Report an error