Terminal Bench 4.1
A benchmark that evaluates the ability of large language models to perform tasks in terminal (command-line) environments, covering scenarios such as file operations, script writing and system command invocation.
Core Facts
Migrating from Gemini 3.8 Flash to Hydrogen, at the default high setting, task completion speed increases 2.5 times, the Terminal-Bench 4.1 score rises from 18.2% to 57%, and fabricated claims decrease by more than 5 times.
Terminal-Bench 4.0 is an evaluation suite for command-line agents, requiring models to complete multi-step operations in real terminal environments, testing coherence and self-correction ability in long-horizon execution.
Timeline (last 90 days)
Different versions of the same benchmark (such as Terminal Bench 4.0 and 4.1) have different problem sets and scoring rules, and scores cannot be compared across versions.
On the same Terminal Bench, Claude Sonnet 5.5 scores 63.9, ahead of Hydrogen's 57%; on computer use tasks Sonnet wins 83.9 to 77.6.
On Terminal Bench 4.1, Gemini 4 Hydrogen reduces fabricated statements by more than 5 times and handles failed tool calls more stably.
On Terminal Bench 4.1, Hydrogen's score rose from 18.2% to 57%, and the proportion of fabricated assertions dropped below 1.5%.
Gemini 4 Hydrogen scored 57% on the Terminal Bench 4.1 test, more than three times the previous generation.
Migrating from Gemini 3.8 Flash to Hydrogen, at the default high setting, task completion speed increases 2.5 times, the Terminal-Bench 4.1 score rises from 18.2% to 57%, and fabricated claims decrease by more than 5 times.
Terminal-Bench 4.0 is an evaluation suite for command-line agents, requiring models to complete multi-step operations in real terminal environments, testing coherence and self-correction ability in long-horizon execution.
All Facts (7)
Migrating from Gemini 3.8 Flash to Hydrogen, at the default high setting, task completion speed increases 2.5 times, the Terminal-Bench 4.1 score rises from 18.2% to 57%, and fabricated claims decrease by more than 5 times.
90%VerifiedTerminal-Bench 4.0 is an evaluation suite for command-line agents, requiring models to complete multi-step operations in real terminal environments, testing coherence and self-correction ability in long-horizon execution.
65%UnverifiedOn the same Terminal Bench, Claude Sonnet 5.5 scores 63.9, ahead of Hydrogen's 57%; on computer use tasks Sonnet wins 83.9 to 77.6.
60%UnverifiedDifferent versions of the same benchmark (such as Terminal Bench 4.0 and 4.1) have different problem sets and scoring rules, and scores cannot be compared across versions.
50%UnverifiedOn Terminal Bench 4.1, Gemini 4 Hydrogen reduces fabricated statements by more than 5 times and handles failed tool calls more stably.
50%UnverifiedOn Terminal Bench 4.1, Hydrogen's score rose from 18.2% to 57%, and the proportion of fabricated assertions dropped below 1.5%.
50%UnverifiedGemini 4 Hydrogen scored 57% on the Terminal Bench 4.1 test, more than three times the previous generation.
50%