Gemini 4 Hydrogen Hands-On: Strong Benchmarks, Questionable Real-World Performance

Gemini 4 Hydrogen is fast and capable, but not a benchmark king—thinking level is the key to success.
Based on the SegFault video and multiple sources, this article evaluates Google's Gemini 4 Hydrogen, released November 12. It's the fast, efficient member of the Gemini 4 family, priced at $1.50/$7.50 per million input/output tokens, supporting multimodal input and roughly a million-token context, with thinking level (low to extra-high) as the core variable. Google reports beating Claude Sonnet 5.5 on 7/10 benchmarks, but test conditions are selective; Artificial Analysis ranks it 16th/80. Going from low to extra-high thinking raises scores from 27 to 54, while cost climbs from 29 cents to $3.16 and time from 1.4 to 14.5 minutes. Plotly's data analysis exam passes 43/43 at about half of Carbon's cost, but it trails on terminal tasks, and GPT 6.1 Sol reaches similar scores at lower cost. Hydrogen is credible for data analysis, tax, and finance agents; start at high level, downgrade for simple tasks, and verify results yourself.
Introduction: Yet Another "Benchmark King"?
On November 12, Google released Gemini 4 Hydrogen. If you build applications with AI, you've probably heard the complaint: Gemini looks great on benchmarks but falls apart in real work. Yet some developers hold the exact opposite view. So, is Hydrogen any good? The YouTube channel SegFault combed through information made public in the two days after launch—not going by gut feeling, but by evidence. They sorted the information into four categories: Google's own claims, measurements from independent tests, capabilities shown in demos, and their own hands-on testing (this last one is still empty, since they haven't run Hydrogen yet).
Based on that video and multiple sources, this article tries to answer one core question: is Hydrogen right for you?
What Hydrogen Is: A Fast, Efficient Middle Tier
Hydrogen is the fast, efficiency-focused member of the Google Gemini 4 family, adapted from Carbon. Carbon and Argon remain Google's deep-reasoning models. On API pricing, Hydrogen costs $1.50 per million input tokens and $7.50 per million output tokens. It supports text, image, audio, and video input, with a context window of about 1 million tokens, shared across what you send, what the model thinks, and what it writes out.
The most critical setting is the "thinking level": low, medium, high, and extra-high. The same model produces wildly different results at different levels. And the thinking process is billed as output—you have to keep this dial in mind at all times.
Google's Own Numbers: Selectively Chosen Comparisons
Google compared Hydrogen against Claude Sonnet 5.5 and GPT 6.1 Tera, both mid-tier competitors rather than top models. Across 10 tests, Hydrogen beat Sonnet on 7. On the coding test SWE Sweep, Google reports a score of 13.5 under its own test harness, versus Sonnet's 3.1 on the public leaderboard. But it also lost 3: 57 vs. 63.9 on Terminal Bench 4.1, 77.6 vs. 83.9 on the OS World computer-use test, and one terminal science test.
The details are worth noting: Google mostly ran Hydrogen at the highest thinking level and used a test harness of its own choosing. Competitor numbers came from a mix of public leaderboards, system cards, and Google's own runs. The OS World score was the best of three runs and based on an offline subset. This is common at launch, but these are claims, not clean comparisons.
Google also says that compared with the previous-generation Gemini 3.8 Flash, Terminal Bench jumped from 18.2 to 57%. If true, that's a huge improvement for the fast tier.

Independent Tests: Strong but Not Top-Tier
Artificial Analysis runs every model through the same nine evaluations. At the extra-high thinking level, Hydrogen scored 54, ranking 16th out of 80 entries (each thinking level is counted separately). Strong, but not top-tier. Claude Fable 5.2 leads with 65 at max level. Google's own Carbon scores 60 at extra-high.
Now turn the dial: low level scores 27, medium 41, high (the default) 51, and extra-high 54. The cost per benchmark task climbs right along with it: 29 cents, 80 cents, $1.87, $3.16. Time per task goes from 1.4 minutes to 14.5 minutes. Why? Writing speed stays almost constant, but at extra-high it writes more—about 300 million output tokens across the whole test suite, roughly four times a typical model.

Those minutes are long benchmark tasks, not chat replies. But the direction is consistent: more thinking, more tokens, more time and money. From high to extra-high, cost rises about 70% for just a 3-point gain.
Another external signal (still marked preliminary): on the Arena voting leaderboard, Hydrogen ranks 15th for text, 10th for web development, and 5th on the agent leaderboard.
Artificial Analysis is an independent AI model evaluation organization whose core method is to run a unified standardized evaluation suite (typically covering reasoning, coding, math, and other benchmarks) on every model and compute a composite index score. Unlike vendor-reported data, it fixes the test harness and prompts, reducing selection bias, and is therefore seen by the community as a relatively neutral third-party signal. But note its limitations: evaluation tasks skew academic and generic, still distant from specific business scenarios; different thinking levels are counted separately, which can dilute or inflate ranking differences; and scores usually carry error margins, so adjacent ranks may not be statistically significant. When reading its rankings, judge alongside task type, cost, and latency rather than looking at rank alone.
Cost-Effectiveness: Competitive, but No Clear Winner
On the small chart on the model page, Hydrogen is the fastest model plotted. But the full table isn't as pretty. Claude Fable 5.2 at low level also scores 54, in under 6 minutes per task, at about the same cost. GPT 6.1 Sol at extra-high scores 52, costing 76 cents.
One clear advantage: on these index tasks, Hydrogen's first answer token arrives in about 17 seconds at high level, versus 43 seconds for Sol at high level. So it's no clear winner on speed or price, but it's competitive.
On your own bill: at list prices (not measured), an agent step sending 200K tokens and returning 20K tokens (including thinking) costs 30 cents input and 15 cents output, 45 cents total. When the same context is sent again, Google's automatic caching takes 95% off the repeated portion. Cache hits are not guaranteed. The output side grows with thinking and never gets cheaper.
One number that doesn't add up: Google's launch post says Hydrogen ties for first at 55.8% on Val's AI legal research benchmark. But Val's public leaderboard (updated November 11) shows Hydrogen at 53.85, ranked sixth. First place is Gemini 4 Carp at 55.77. We don't know why the discrepancy exists; neither page explains it. So we can't call Hydrogen number one, but sixth isn't weak either: each score has an error margin of about 3.5 points, and the top six are within 2 points of each other. It's close to the leaders, with no clear gap.

Another independent test: charting company Plotly's data analysis exam, 43 tasks over a data warehouse of about 20 million rows, including traps like ambiguous questions and impossible requests. Hydrogen passed all 43 tasks, Carbon 42, Argon 41—close to a tie by Plotly's own rules. But Hydrogen ran at about half the cost, $6.36. GPT 6.1 Sol also passed all 43, costing $2.69, with a median time of 18 seconds per question versus 32 seconds for Hydrogen. Two other models also scored 43. Plotly states clearly: each model was run only once, and repeat runs can swing scores by up to 3 points. Their exam, their harness, thinking level unspecified. Useful, but just one data point.
Automatic prompt caching is a discount mechanism API providers offer for repeated context: when multiple requests share the same prefix content (such as a system prompt, long document, or conversation history), the server caches the computed result for that part, and subsequent input tokens that hit the cache are billed at a steep discount. The 95% discount mentioned here is one such case. Its key limitation is that "hits are not guaranteed"—caches have a time-to-live, a minimum token threshold, and routing consistency requirements, and long gaps between requests or load changes can cause misses. So when estimating costs, treat caching as a potential optimization rather than a guaranteed saving, and don't count on any discount for the output side (especially thinking tokens).
Demos and Early Reviews: Single Runs, Limited Signal
Google showed Hydrogen generating a store page with live shaders in one shot, turning Google Sheets into an animated pixel canvas, making slides and fixing its own text overlap. Demos show what's possible, but they're picked by the people selling the model and don't tell you the success rate. Some demos had extra help: the store page used Google's front-end design skill rather than the bare model.
Outside Google, the first batch of reviews are all single runs. Analytics Vidya, at medium thinking, had it write a SQL query that passed all 13 test cases, but also cut a video scene a second or so early. Active Pieces (an article written with an automated research system) gave it three simple automation tasks, all correct, same as the much cheaper GPT-5.4 Mini. But at default thinking, median response time was 4.4 seconds, versus 0.9 seconds for the latter.
Does Gemini Really Fail at Real Work?
The record does have problems. In 2024, Google paused Gemini's ability to generate images of people. In 2025, Ars Technica reported that Gemini CLI wiped a user's files. In September, AI Weekly cited Bloomberg reporting that Google employees said an unreleased Gemini 4 model performed better on benchmarks than on real coding—that wasn't about Hydrogen, and Google called the report inaccurate.

The same record has highlights: in 2024 Gemini pioneered million-token context, and in 2025 Gemini 2.5 Pro topped LM Arena. The fair characterization is "inconsistent," not "bad." Two days is too short to say Hydrogen has solved the problem.
Google's own model card reports: in automated safety testing, unjustified refusals rose about 5.5 percentage points over 3.8 Flash, especially on multimodal prompts. This is lab testing, not how often you'll be refused, but Google chose to report it.
A model card is standardized documentation shipped with a model release, describing intended use, training and evaluation methods, safety test results, and known limitations. Unlike marketing material, it's usually written by the team responsible for model development and includes quantitative metrics such as refusal rates, harmful content tests, and bias evaluations. The "unjustified refusals up about 5.5 percentage points" cited here comes from such documentation. When reading model cards, note: these are measurements in controlled lab environments, and the prompt distribution differs from real user requests, so the numbers are better used for cross-model comparison and risk signaling than for predicting your specific probability of being refused in production.
Should You Use It?
The evidence points to the following conclusions:
- Data analysis, tax, and finance agents: Hydrogen is a credible choice. Both Val's (tested at extra-high) and Plotly place it at or near the top.
- Terminal-heavy agents: Look at other options too. Google's own table shows it trailing Sonnet on Terminal Bench, and Val's ranks it 11th/64 on Terminal Bench 4.0.
- Don't default to extra-high: Google itself warns it may overthink simple prompts. Start at high. For simple steps like routing or extraction, try medium or low, and check the results yourself.
- Try GPT 6.1 Sol: On Artificial Analysis and the Plotly exam, it reaches similar scores at lower cost.
Two caveats when switching: the minimum thinking level now returns an error; the old thinking budget on Google Cloud also errors; and the temperature parameter is ignored.
Hydrogen hasn't ended the Gemini debate. It's a fast, capable model with a dial you have to set yourself. That empty drawer? Next, SegFault wants to fill it: the same real tasks compared across models, with cost and time on screen.
Key Takeaways
- Hydrogen is the fast, efficient model in the Gemini 4 family, priced at $1.50/$7.50 per million input/output tokens, with a million-token context, and thinking level is the key variable.
- Google's own benchmarks show it beating Claude Sonnet 5.5 (7/10 tests), but the test conditions are selective; in independent tests, Artificial Analysis ranks it 16th/80, not top-tier.
- From low to extra-high thinking, scores rise from 27 to 54, but cost and latency rise sharply (29 cents → $3.16, 1.4 minutes → 14.5 minutes).
- Performance is credible on data analysis, tax, and finance agent tasks (Plotly 43/43), but it trails on terminal tasks, and its cost-effectiveness versus GPT 6.1 Sol isn't outstanding.
- Start at high level, use medium/low for simple tasks, and verify results yourself; when switching, note limitations like errors at the minimum thinking level.
One-line Summary
Gemini 4 Hydrogen is fast and capable, but it's no benchmark king—thinking level is the make-or-break factor.
Paragraph Summary
Based on the SegFault video and multiple sources, this article evaluates Google's Gemini 4 Hydrogen, released November 12. It's the fast, efficient member of the Gemini 4 family, priced at $1.50/$7.50 per million input/output tokens, supporting multimodal input and roughly a million-token context, with thinking level (low to extra-high) as the core variable. Google reports beating Claude Sonnet 5.5 on 7/10 benchmarks, but test conditions are selective; in independent tests, Artificial Analysis ranks it 16th/80, with mid-to-upper placements across Arena leaderboards. Going from low to extra-high thinking raises scores from 27 to 54, while cost climbs from 29 cents to $3.16 and time from 1.4 to 14.5 minutes—high to extra-high costs about 70% more for just 3 points. It passes Plotly's data analysis exam 43/43 at about half of Carbon's cost, but trails on terminal tasks, and GPT 6.1 Sol reaches similar scores at lower cost. The conclusion: Hydrogen is credible for data analysis, tax, and finance agents; start at high level, downgrade for simple tasks, and verify results yourself.
Related articles

The 15-Year Programmer Who's Been Unemployed for Two Years: The Tech Worker's Dilemma in the AI Era
A US senior software engineer with 15 years of experience has been unemployed for two years, exhausted his savings, and sought help on Hacker News. This article analyzes the structural causes of long-term unemployment among tech workers, strategies for addressing resume gaps, and the redefinition of experience value in the AI era.

Soundcore Liberty 5 Pro Drops to All-Time Low: The Go-To True Wireless Earbuds for Commutes and Calls
Soundcore Liberty 5 Pro drops to an all-time low of $129.99 (from $169.99). These Anker true wireless earbuds excel at call quality and are on sale at Amazon, Best Buy, and Walmart in four colors—ideal for commutes and online meetings.

After Anthropic's IPO, How Far Can Safety-First Go?
The Verge podcast explores Anthropic's post-IPO core tension: can a public company stay safety-first? How public market pressure reshapes AI lab safety decisions.