rsi.watch

How we rate evidence

Each entry answers two separate questions. Is the record right: did this person or lab say this, on this day, in these words? And how strong is the evidence for what they said?

Is the record right

This one is not a rating. We keep a text copy of the source page, and the site will not build unless every quote, and every number and date in our own notes, appears in that copy. An entry either passes or it is not published. The entry shows the day we checked it.

How strong is the evidence

This one is a rating, and it comes in four levels. It is adapted from GRADE, the framework medical reviews use to rate how certain an estimate is; what we kept and what we changed is below. We rate a single claim, not a model and not a person.

  • HighWe are confident the claim holds as stated.
  • ModerateThe claim probably holds, but a better test could change it.
  • LowThe claim may be substantially wrong.
  • Very lowThe evidence tells us little about whether the claim holds.

Where a rating starts

High
Someone other than the developer measured it.
Moderate
The developer's own paper, tested on tasks held out from the improvement, with public code.
Low
The developer's own numbers, in a paper or a post.

What lowers it, one level each

Risk of bias
The party with a stake controls the measurement: a model grades its own work, the developer edits the outside report, or the score reported is the score the system was optimized for.

For example, Claude Opus 5.5 evaluation: The developer of the model could edit the outside assessment before it was published.

For example, R&D Automation Index: Anthropic measured its own work, and a Claude model graded how much Claude did. Staff ratings were within one level of the judge 97% of the time.

Inconsistent results
The source's own numbers disagree, or other primary sources give different figures for the same thing.
Indirect measure
What was measured stands in for the claim: a subset of a benchmark, or code merged as a stand-in for research done.

For example, InnovationEval: One post-training task, scored against one paper, stands in for AI R&D as a whole.

For example, Research acceleration: The view inside OpenAI: Agent usage, agent-workdays and experiment counts stand in for progress toward an automated researcher.

Imprecise
Too few tasks, runs or rounds to rule out chance, or no stated uncertainty.

For example, MASS: The benchmark gains come from two rounds on one base model, and the page gives no uncertainty for the ratios.

For example, InnovationEval: Each model was evaluated once, and Epoch notes that run-to-run variability could change the result.

Selective reporting
Only demos, or only the runs that worked.

What raises it, one level each

Independently replicated
Someone else ran it again and got a similar result.
Others can rerun it
Code and weights are public, so anyone can check.

What we do not rate

A statement of position or a forecast has no measured result behind it, so there is nothing to rate. Those entries say "not rated" and why.

Every reason is quoted

Each step down or up on an entry cites a line from the source, and the build checks that line against the saved copy, the same way it checks quotes. Many of the reasons are the source's own stated limits.

Very low is the floor, so a rating can stop moving before the list of problems ends. We still list every serious limit the source admits, and when the source answers a limit in the same passage, we say that too.

What we do not do

We do not count how many sources say yes. One well-measured result tells you more than ten announcements, and counting rewards whoever publishes most.

Where this method comes from

The rating is adapted from GRADE (Grading of Recommendations, Assessment, Development and Evaluation), the framework that systematic reviews and clinical guidelines use to say how certain an estimate is. It is set out in Guyatt et al., “GRADE: an emerging consensus on rating quality of evidence and strength of recommendations”, BMJ 2008;336:924, and in the GRADE Handbook.

What we kept

  • Four levels of certainty, from high to very low.
  • A starting level set by how the evidence was produced, then steps down for specific problems and up for specific strengths.
  • Four of the five reasons for rating down: risk of bias, inconsistency, indirectness and imprecision. Publication bias becomes selective reporting.
  • A stated reason for every step, so a reader can disagree with a specific step instead of with a label.

What we changed, and why

  • The starting point. GRADE starts randomized trials high and observational studies low. AI claims are rarely trials, so we start from who did the measuring: an outside party, the developer with held-out tests and public code, or the developer alone.
  • The reasons for rating up. GRADE rates up for a large effect, a dose-response gradient, or confounding that would work against the result. Those rarely apply here. We rate up for independent replication, and for public code and weights that let anyone rerun the result.
  • One claim, not a body of evidence. GRADE rates all the studies on one question together. We rate the evidence behind one statement by one lab or person.
  • One step per reason. GRADE allows two steps down for a very serious problem. We use one step per reason, so ratings stay easy to check.

11 entries are rated so far. The rest show their source type until they are rated.