Terminal-Bench
An AI evaluation benchmark focused on real-world software engineering tasks in command-line terminal environments. It requires agents to complete end-to-end engineering tasks via the command line—such as installing dependencies, running scripts, and debugging errors—assessing an AI's ability to execute tasks independently without continuous human intervention.
Core Facts
Terminal-Bench is a benchmark specifically designed to evaluate AI models' ability to execute complex tasks in command-line environments, covering scenarios such as file operations, script writing, and system debugging.
On the SWE-Bench Pro benchmark, Sonnet 5 significantly trails Opus 4.8.
Timeline (last 90 days)
Anthropic found that under strict per-task CPU and memory limits, as many as 6% of Terminal-Bench tasks failed due to pod errors unrelated to the model; after loosening the limits, scores moved by up to 6 percentage points.
On Terminal-Bench 4.0, Haiku 5.5 scored 39.2 and GPT-6 Luna scored 16.4.
In both the DeepSWE and Terminal-Bench evaluations, Replit Agent achieved Pareto optimality relative to Astra running alone, with no published Astra baseline achieving both lower cost and higher score.
The paper explicitly acknowledges that the Terminal-Bench pilot does not support a superiority conclusion, as the reliability layer delivered no better success rate at higher cost.
In the Terminal-Bench Core 0.1.1 pilot across 12 selected tasks, both the baseline group and the reliability-layer group each passed 17/36 under strict trials, with the reliability layer consuming 37.49% more input tokens and 50.73% more output tokens.
After fine-tuning on agent trajectories, CLM achieved 87.6% on Terminal-Bench 2.1 and 81.6% on DeepSWE.
Terminal-Bench focuses on the ability to complete multi-step command-line tasks in a terminal environment, while SWE-Atlas (and its predecessor SWE-bench) focuses on the end-to-end ability to locate and fix issues in real GitHub repositories.
The original Terminal-Bench focuses on command-line operation evaluation.
Terminal-Bench-Science is the science branch of the Terminal-Bench family.
SWE-2 achieved a score of 92.8 on the Terminal-Bench 2.1 benchmark.
5 more timeline events
All Facts (14)
Terminal-Bench is a benchmark specifically designed to evaluate AI models' ability to execute complex tasks in command-line environments, covering scenarios such as file operations, script writing, and system debugging.
90%VerifiedOn the SWE-Bench Pro benchmark, Sonnet 5 significantly trails Opus 4.8.
65%UnverifiedTerminal-Bench requires models to perform multi-turn interactions in a real shell environment, including issuing commands, parsing stdout/stderr output, handling permission issues, and managing filesystem state.
60%UnverifiedAnthropic found that under strict per-task CPU and memory limits, as many as 6% of Terminal-Bench tasks failed due to pod errors unrelated to the model; after loosening the limits, scores moved by up to 6 percentage points.
50%UnverifiedOn Terminal-Bench 4.0, Haiku 5.5 scored 39.2 and GPT-6 Luna scored 16.4.
50%UnverifiedIn both the DeepSWE and Terminal-Bench evaluations, Replit Agent achieved Pareto optimality relative to Astra running alone, with no published Astra baseline achieving both lower cost and higher score.
50%UnverifiedIn the Terminal-Bench Core 0.1.1 pilot across 12 selected tasks, both the baseline group and the reliability-layer group each passed 17/36 under strict trials, with the reliability layer consuming 37.49% more input tokens and 50.73% more output tokens.
50%UnverifiedThe paper explicitly acknowledges that the Terminal-Bench pilot does not support a superiority conclusion, as the reliability layer delivered no better success rate at higher cost.
50%UnverifiedAfter fine-tuning on agent trajectories, CLM achieved 87.6% on Terminal-Bench 2.1 and 81.6% on DeepSWE.
50%UnverifiedTerminal-Bench focuses on the ability to complete multi-step command-line tasks in a terminal environment, while SWE-Atlas (and its predecessor SWE-bench) focuses on the end-to-end ability to locate and fix issues in real GitHub repositories.
50%UnverifiedThe original Terminal-Bench focuses on command-line operation evaluation.
50%UnverifiedTerminal-Bench-Science is the science branch of the Terminal-Bench family.
50%UnverifiedQwen3 27B's performance on coding and agent benchmarks is: SWE-bench Pro 61.7, Terminal-Bench 2.1 73.0, and LiveCodeBench V6 90.3.
50%UnverifiedOn Terminal-Bench 4.0, the standard version of Claude 5.1 scored 55.8% and the trusted version reached 60.9%; the OS World 2.0 partial completion rate was 77.9%.
50%