KernelBench

2025-02-14 · Anne Ouyang et al.

What KernelBench measures, in its makers' own words, and the claim on the rsi.watch ledger that cites it. We do not list scores; its own page has those.

What it measures: “A benchmark and environment for evaluating LLMs' ability to generate efficient GPU kernels”
Tasks: “We introduce KernelBench, an open-source framework for evaluating LMs' ability to write fast and correct kernels on a suite of 250 carefully selected PyTorch ML workloads.”
How it is scored: “We introduce a new evaluation metric fast_p, which measures the percentage of generated kernels that are functionally correct and offer a speedup greater than an adjustable threshold p over baseline.”

Limits its makers state

“We provide some reference baseline times for a variety of NVIDIA GPUs across generations in results/timing, but we recommend you generate your own baseline time for more accurate results (cluster power, software version, all affect timing results).”
“The KernelBench team does not review, validate, or endorse individual kernels or reported results. Users are responsible for independently verifying any results obtained using the framework.”

Before comparing numbers

Weco's 55% to 32% and 63% to 34% are reward-hacking rates that Weco measured itself on a set of KernelBench tasks. They are not fast_p scores.
The task count and the fast_p definition come from the arXiv v1 abstract of 14 Feb 2025. The repo says it continues to be updated, and its Hugging Face dataset has moved to v0.1.
The repo lists Level 1 and Level 2 with 100 problems each, Level 3 with 50, and a Level 4 of Hugging Face models with no count given.

On the ledger

Weco AI: AIDE² 2026-09-22Reward hacking, which the paper measures on "a subset of KernelBench", "falls from 55% to 32% during the run" (paper).Weco's blog reports the same kind of rate on its set of KernelBench tasks as 63% to 34%.

Date of the arXiv v1 abstract page.

source paper

Other benchmarks the ledger cites Aider Polyglot, RSI-Inference Bench, SWE-bench Verified, Terminal-Bench 2.0

← All benchmarks