Gemini 4 Argon Hands-On: Strong Benchmarks, Alarming Code Audit Failure Rate

Google has finally released Gemini 4 Argon — marking a full eight-month gap since the last major Pro-series model update. After getting early access, a YouTube tech creator ran a deep round of testing with real engineering projects, and the conclusions are far more nuanced than the official benchmark numbers suggest.

Argon Arrives After an Eight-Month Delay

According to the creator, the gap between Gemini 3.1 Pro and Gemini 4 Argon spanned eight months. During that time, Google only updated the Flash-series models. Argon is now available via the API, Google AI Studio, the Gemini platform, Anti-Gravity, and the Gemini App for Ultra and Pro subscribers.

Google positions Argon as excelling at "deep, complex real-world tasks across domains," covering programming, finance, law, cybersecurity, and scientific research. The company claims it ranks first on the VALS index, which measures model performance in key economic domains such as finance, programming, law, and tax.

There's an important pricing detail to note: the current API pricing is $2 per million input tokens and $10 per million output tokens, but this is introductory pricing. Starting December 1, it will double to $4 per million input tokens and $20 per million output tokens.

The Gap Between Benchmarks and Real Experience

On the Artificial Analysis Intelligence Index, Argon scores higher than Astra and 6.1 Sol, but noticeably lower than Opus 5.5's high-reasoning mode. More critically, due to poor token efficiency, Argon's actual cost of use is higher than the higher-scoring Opus 5.5.

The creator specifically flagged a frustrating issue: Argon's cache hit rate is terrible, only 70%–80%, while other models typically achieve over 99%. This means the 95% cache read discount doesn't apply, significantly increasing actual input costs.

On hallucination control, Argon recorded a 15% hallucination rate on Artificial Analysis's AI Omniscience benchmark — the lowest among all models scoring above 45 on the intelligence index. However, that benchmark also factors in incorrect answers, partial answers, and refusals — when looking at the combined score, Argon drops to fourth place. Because it doesn't answer enough questions correctly, a strategy of "refusing to answer to avoid errors" can't compensate for insufficient accuracy.

Cache hit rate refers to the proportion of repeated or similar prefix tokens during LLM inference that can be read directly from cache without recomputation. Mainstream commercial APIs typically offer discounts of over 90% on cache reads (for example, Anthropic charges a 25% premium for cache writes and offers a 90% discount on reads). When the cache hit rate stays above 99%, input costs for multi-turn conversations or long-context tasks can be dramatically reduced. Once it drops to 70%–80%, it means roughly 20%–30% of input tokens must be recomputed at full price, with the 95% discount applying only to the cached portion. For agent scenarios that repeatedly reference large codebases, this gap gets amplified across turns, directly inflating the actual bill per task.

The Artificial Analysis Intelligence Index is a comprehensive third-party benchmark that weights and aggregates scores from reasoning, programming, math, science, and other evaluations into a single comparable number, often used as a rough measure of a model's general intelligence. One known limitation: it measures point-in-time capability, not token efficiency or output per unit cost. This explains why Argon can score higher than Astra overall yet lower than Opus's high-reasoning mode, while actually costing more to use — a high score doesn't equal high cost-effectiveness, which is a common source of the gap between leaderboard rankings and real-world engineering.

Code Auditing: The Most Brutal Testing Ground

The creator designed a code audit benchmark where models review real project codebases, and then every claim is verified for accuracy. Results across three projects:

Project One (Tricer compiler Rust rewrite): 6.1 Sol scored 8.84, with all 38 claims correct; Opus 5.5 scored 8.78, with only 1 error; Gemini 4 Argon scored 4.38, with 24 false claims — most of its descriptions of the codebase were wrong.

Project Two (T3 Code remote update workflow audit): Opus 5.5 scored 8.22; 6.1 Sol had no false claims and took only 10 minutes and $1; Gemini 4 Argon again performed poorly.

Project Three (Lakebed product audit): 6.1 Sol scored 9/10; Gemini 4 Argon scored only 2.25/10 — 56 of 78 findings were wrong, a failure rate of 70%.

The creator put it bluntly: "This model is more likely to gaslight your agent than give useful feedback."

Code audit benchmarks are fundamentally different from typical code generation evaluations. They don't ask the model to write working code; instead, they require the model to read a real codebase and make verifiable claims about architecture, dependencies, potential defects, and so on — then a human checks each claim for accuracy. This approach directly exposes a model's "hallucinated confidence": it may fluently describe a function that doesn't exist, misjudge data flow direction, or mistake an irrelevant module for core logic. Because every claim must be verified, the false positive rate becomes glaringly obvious — it measures whether the model "says the right thing," not whether it "writes something plausible."

Frontend Design and 3D Capabilities: Two Sides

On frontend design, Argon performs reasonably well. The homepage it generated, per the creator, "if you stumbled across it, you wouldn't guess it was AI-generated." It works noticeably better when paired with Anthropic's design skill; turn the skill off and design quality drops sharply. But it still falls short of Fable 5.1.

The 3D capability test (Fish Slop game) exposed a classic "Gemini-style" problem: the fish animation and eye direction were handled well, but there were laughably absurd logic flaws — food at the bottom of the fish tank never expires, and after swimming over, the fish can't eat it, so it just stares at the food until it starves. Another example: monsters refresh every 60 seconds, and killing them drops a buff item lasting 30 seconds, but monsters don't appear while the buff is active, making the item completely useless.

The creator summarized these issues as the model lacking "big model smell" — that is, insufficient deep understanding of the system as a whole. It can write code that looks reasonable but can't judge whether its logic is internally consistent.

The SQLite Fix Disaster

In a real SQLite connection stability fix task, the creator already knew the correct approach was a connection pool with single-writer, multiple-reader. Argon correctly identified the read-write separation direction, but generated a PR of 1,200 lines (should have been 30), took 30 minutes, completely violated the repo's formatting standards, failed lint checks, and didn't address issues raised by the review bot.

Ultimately, Opus 5.5 reviewed and confirmed the PR was invalid and closed it — the actual correct fix required only 21 lines of code. Argon's problem: it can read code and explain how it works, but never tests its own theories or checks whether existing PRs already contain the correct solution.

Unique Platform Advantages and Real Limitations

Argon isn't without value. Google's YouTube API access for Gemini models is very generous, and the Gemini API is currently one of the best ways to get transcripts from YouTube videos. For YouTubers or those working with video, this is a genuine advantage. Historically, Gemini models have also performed well on visual tasks, such as identifying motherboard components and annotating them with Python.

But the creator also admitted that Gemini models actually offer a poor experience when using Google's own services (like email). And the Anti-Gravity integration process was extremely painful — model slugs are forcibly bound to reasoning levels, connections fail mid-run with no history recovery, and numerous bugs require developers to work around them manually.

Anti-Gravity is Google's developer-facing agentic coding environment that allows models to autonomously execute multi-step tasks, call tools, and manipulate files. The "model slug and reasoning level forcibly bound" mentioned in the article means developers can't independently choose model version and reasoning intensity, only accept preset combinations; "connections fail mid-run with no history recovery" reflects immature session state management. Such integration obstacles significantly raise the cost of use: teams not only bear the model's error rate but also invest extra engineering effort to work around platform-level defects — a practical reason many developers abandon the toolchain at the evaluation stage.

A Developer's Final Decision

The creator made a clear decision: PRs generated by Gemini 4 Argon and the standard Muse Wonder are banned from merging into the T3 Code codebase. He plans to implement an automatic closure mechanism for such PRs within the week.

"The time and tokens I spend reviewing code written by Gemini 4 exceed what it would take to just use a smarter model to write the code directly," he said, adding that aside from using Gemini to fetch YouTube data, he won't use it for anything else.

Was that too harsh? The creator asked himself the same question. But considering the code audit data he obtained across three independent projects — with failure rates ranging from 50% to 70% — this is clearly not an emotional judgment.

For teams currently evaluating whether to incorporate Argon into their development workflow, the recommendation is to at least run an audit test using your own project's real code — don't make decisions based solely on official benchmark data.