TerminalBench
An AI capability evaluation benchmark used to assess performance on terminal-based tasks.
Core Facts
Timeline (last 90 days)
The blogger pointed out four discount factors in the benchmark comparison: different test frameworks (TerminalBench 1017 ran within the ClaudeCode framework while the baseline ran within the DeepSeek framework), Artificial Analysis's own test of the old Flash on TerminalBench scored only 26.8% rather than 31.2%, the competitor comparison used Opus 5 and GPT 5.6 Sol rather than the latest models, and the old model's NL2 repo score is inconsistent across different tables (64.0 vs 65.4).
Opus 5.5 scores 66.4% on TerminalBench, about 24 points higher than DeepSeek 1017; GPT 6.1 Sol leads 1017 slightly on DeepSWE at 75.2%.
On TerminalBench, HydraFusion has the same cost as GPT-5.6 Sol but achieves 11.4 points higher quality.
Most mainstream agent evaluation benchmarks (such as Web Arena, SWE-Bench, Gaia, and TerminalBench) assume a static environment in which the environment does not change during evaluation.
On the SWE-Bench Pro benchmark, Sonnet 5 significantly trails Opus 4.8.
All Facts (5)
On the SWE-Bench Pro benchmark, Sonnet 5 significantly trails Opus 4.8.
65%UnverifiedThe blogger pointed out four discount factors in the benchmark comparison: different test frameworks (TerminalBench 1017 ran within the ClaudeCode framework while the baseline ran within the DeepSeek framework), Artificial Analysis's own test of the old Flash on TerminalBench scored only 26.8% rather than 31.2%, the competitor comparison used Opus 5 and GPT 5.6 Sol rather than the latest models, and the old model's NL2 repo score is inconsistent across different tables (64.0 vs 65.4).
50%UnverifiedOpus 5.5 scores 66.4% on TerminalBench, about 24 points higher than DeepSeek 1017; GPT 6.1 Sol leads 1017 slightly on DeepSWE at 75.2%.
50%UnverifiedOn TerminalBench, HydraFusion has the same cost as GPT-5.6 Sol but achieves 11.4 points higher quality.
50%UnverifiedMost mainstream agent evaluation benchmarks (such as Web Arena, SWE-Bench, Gaia, and TerminalBench) assume a static environment in which the environment does not change during evaluation.
50%