Benchmarks / Terminal-Bench 2.0
Terminal-Bench 2.0
What Terminal-Bench 2.0 measures, in its makers' own words, and the claim on the rsi.watch ledger that cites it. We do not list scores; its own page has those.
- What it measures
- “a harder, better verified version of Terminal-Bench and a new package for evaluating and optimizing agents.”
- Tasks
- “We're releasing Terminal-Bench 2.1 to fix issues in 28 of the 89 tasks in Terminal-Bench 2.0.”
Limits its makers state
- “Terminal-Bench 2.0 pinned pre-built Docker images for reproducibility, but internet access introduces external dependencies that can change over time. We identified nine tasks where external dependencies changed after the benchmark was built.”
- “Eight tasks had insufficient resource budgets for at least one valid approach (either the oracle solution or a frontier model solution) to finish consistently.”
- “Some tasks had instructions that were not aligned to their tests.”
Before comparing numbers
- The makers' news page describes the first Terminal-Bench as "An evaluation framework and benchmark for agents in the terminal".
- The makers' posts we saved give no scoring rule for 2.0.
- On May 6, 2026 the makers released Terminal-Bench 2.1, which fixes issues in 28 of the 89 tasks in 2.0. Sakana MASS reports its score-per-output-token ratio on 2.0, and scores on 2.0 and 2.1 come from different versions of those 28 tasks.
On the ledger
- Sakana AI: MASS 2026-10-12Score per output token is 0.96× the base value on Terminal-Bench 2.0 and 0.94× on SWE-bench Verified
Date listed for the Terminal-Bench 2.0 and Harbor post on the tbench.ai news page. The post itself shows no date.
Other benchmarks the ledger cites Aider Polyglot, KernelBench, RSI-Inference Bench, SWE-bench Verified