Self-improving AI / Self-Rewarding Language Models
Self-Rewarding Language Models
“In this work, we study Self-Rewarding Language Models, where the language model itself is used via LLM-as-a-Judge prompting to provide its own rewards during training.”
Self-Rewarding Language Models is one of the projects on the self-improving AI map. Below is what it changes about itself, whether the change lasts, and whether it improves the improver, each with the line from its own source.
Where it sits
Columns: what it changes, from the surface to the deepest change. Rows: whether the change outlasts the run. A number is how many projects sit in that square. See it on the 3D map
- What it changes
- Its model weights“A preference dataset is built from the generated data, and the next iteration of the model is trained via DPO.”
- Does the change last
- Becomes the next version“Our overall self-alignment procedure is an iterative one, which proceeds by building a series of such models, with the aim that each improves over the last.”
- Does it improve the improver
- improves the improver“We find that performing a round of self-reward training improves the ability of the model at providing self-rewards for the next iteration, in addition to its improved instruction following ability.”
- Who sets the goals
- people set the aim, the system picks tasks“We generate a new prompt x_{i} using few-shot prompting, sampling prompts from the original seed IFT data”
- How it was tested
- tested on held-out tasks“We evaluate the LLM-as-a-Judge via various metrics which measure alignment with held-out human preference data.”
A fixed Llama 2-Chat 70B writes new prompts from the seed data; the model being trained (Llama 2 70B) answers them, scores its own answers through an LLM-as-a-Judge prompt, and is trained on the resulting preference pairs with DPO. Each model makes the training data for the next. The judge is the same model, and its pairwise agreement with held-out human rankings went from 78.7% to 80.4% to 81.7% across the iterations. Humans supply the seed data and the judging prompt; the authors call the results preliminary and ran three iterations in a single setting.
Does it count as RSI?
It depends on the definition. Under the four used on What counts as RSI, by what its own source says:
- Loose (Eliezer Yudkowsky, Intelligence Explosion Microeconomics (2013)): counts
- Durable (Ren et al., self-improvement survey (2026)): counts
- Strict (Roman Yampolskiy, From Seed AI to Technological Singularity (2015)): counts
- Autonomous (Anthropic, When AI builds itself): does not count
How it got here
Earlier work in this catalog that Self-Rewarding Language Models cites, oldest first. Under each: where that work sits on the map, then where Self-Rewarding Language Models sits (changes in bold), and what Self-Rewarding Language Models' own paper says about it. Its whole family on the citation map
It cites nothing in this catalog, or our copy has no reference list.
Cited by Ren, self-improvement 2026, Recursive Harness Self-Improvement 2026, SPADE 2026, Harness² 2026, LM2LM 2026, Self-HPO 2026, Pu, recursive self-improvement 2026
In the same square
CoEvo-RSI, ECG-RSI, MTA-Zero, R²Orch, Sakana MASS (Multi-Agent Self-Supervision), Self-HPO, SPADE, STaR: Self-Taught Reasoner, Terminal-RSI