[KongchangAI]
· 3 min read· 1,551 words

Is Gemini 4 Neon Really Better Than GPT-6.2 Sol? Breaking Down the Benchmarks

Is Gemini 4 Neon Really Better Than GPT-6.2 Sol? Breaking Down the Benchmarks

Gemini 4 Neon's 16 official first-place finishes shrink under independent re-testing; performance slightly higher but cost and speed clearly worse.

The article breaks down Google's Gemini 4 launch chart where Neon took first in 16 of 19 tests, noting that competitor scores were cobbled together and the strongest rival Claude Fable was omitted. Independent evaluator Valse re-ranked Neon second, less than half a point behind Fable. The four-fold bug-finding advantage stemmed from Google's custom tool; with a unified tool the gap shrinks to about one percentage point. The cost scatter plot shows Neon's top setting scores 61.4, only about 4 points above Sol, at six times the cost and about 21 minutes per run. In five real-world tests Neon passed 4 challenges for about $1.50 total but was often slower and pricier. Conclusion: subscribers should use Neon for complex code, while API payers should prefer cheaper models for daily tasks.

The Real Gap Behind the Official Blue Table

When Google released the Gemini 4 series, a comparison chart covering 19 benchmarks showed Neon taking first place in 16 of them, and the chart racked up nearly 800 upvotes on tech forums overnight. But look closely at that table: most of the competitors' scores weren't actually run by Google—they were cobbled together from various vendors and review sites. More importantly, Google only included the cheaper version of GPT-6.2 Sol and an older version of Claude, completely omitting its strongest rival, Claude Fable. After independent evaluator Valse re-ranked the results, Claude Fable took first place at 71.31%, with Neon second at 70.87%—a gap of less than half a point.

After re-ranking by independent evaluator Valse

In the same table, on the Chartography chart-reading test, Neon scored only 73.5%, losing to GPT-6.2 Sol's 75.1%; and the winner of both coding tests became Claude. The halo of the official chart narrows quickly in the face of independent data.

The Truth Behind the Four-Fold Advantage: Tool Wrapper or Model Capability?

The most eye-catching reversal is hidden in the "find program bugs" test: on Google's big table, Neon finds 39 out of every 100 code bugs, while GPT-6.2 Sol finds only 8.3—a four-fold gap that looks like a crushing victory. But after AI blogger Mervin Prezen dug into the test conditions, he found that each model was using a completely different auxiliary application—Google gave Neon its own custom tool built specifically for bug-finding, the equivalent of handing your opponent a manual screwdriver while equipping yourself with a power impact drill.

When re-tested with the exact same auxiliary tool, Mini SWE, Neon's score dropped from 39% straight down to 12.8%, while GPT-6.2 Sol scored 11.6%—suddenly shrinking to a close hand-to-hand fight with only about a one-percentage-point difference. Mervin's conclusion is blunt: Neon is an application, not an artificial intelligence; the four-fold crushing on the table is mostly thanks to the external tool wrapper, not because Neon itself is four times stronger than Sol.

Not because Neon itself is four times stronger than Soul

The difference between AI agents and underlying models: The recurring distinction between "application" and "artificial intelligence" in this article refers to the layering of AI agents and base large models. Base large models (such as GPT-6.2 Sol and Claude Fable) are only responsible for understanding and generation; an AI agent, on the other hand, wraps another layer of engineering framework around the model, including tool calling, multi-step planning, code execution sandboxes, error retries, and other modules. An agent's performance on benchmarks equals the product of "model capability × framework capability," so the same model with different frameworks can differ in score by several times. Google's custom bug-finding tool for Neon is essentially a reinforced agent scaffold, which explains why the gap shrinks dramatically once the tool is unified.

The Brutal Conversion of Performance and Cost

The capability-cost scatter plot from independent organization Artificial Analysis provides a more concrete ledger. The further right on the horizontal axis, the more expensive; the further up on the vertical axis, the stronger. Neon appears as four points on the chart because its thinking time is adjustable—the longer it thinks, the higher the score. The leftmost gray point represents GPT-6.2 Sol, scoring 57 at about $1 per test; Neon at its highest thinking setting scores 61.4, about 4 points more than Sol, but the cost per test soars to $6.44—six times that of Sol—and each test run requires waiting about 21 minutes, marked as "very slow."

This cost chart also omits Claude Fable and another version of GPT-6.2 called Astra, both of which rank ahead of Neon. Astra scores slightly higher than Neon and costs only $4 per test—cheaper than Neon's highest setting. Another developer had both draw the same image: Google's model took almost twice as long as its opponent, cost $4, while the opponent cost only $2.50.

The opponent cost only $2.50

Next to Google's own price list hides a small asterisk that could easily make you waste money: until November 30, the promotional unit price of the larger flagship Carbon is actually lower than that of the mid-sized Neon. If you don't read the asterisk carefully before ordering, you could very well pay more for a medium cup.

Thinking time (reasoning budget) and the cost-performance trade-off: Neon's adjustable "thinking time" corresponds to the test-time compute scaling mechanism—the model performs longer internal reasoning chains or multiple samplings before generating an answer, thereby improving scores on complex tasks. Such mechanisms typically make accuracy grow logarithmically with thinking time, i.e., diminishing marginal returns. Therefore, the highest setting's score of 61.4 offers only limited improvement over lower settings, while costs multiply. Artificial Analysis's scatter plot visualizes exactly this trade-off: different settings correspond to different coordinate points, and users need to choose a setting based on the value of the task rather than defaulting to maximum.

Security Tests and Real-World Bills

In five cybersecurity tests, Neon scored 76% on secure building, higher than Claude Fable's 68%; it scored 90% on internal vulnerability discovery, 2 percentage points higher than Sol; but it lost to its own larger sibling Carbon when verifying whether vulnerabilities are real, and lost to Claude Fable on vulnerability fixing by 71 to 75. These scores all come from the agent version Neon for Cyber with security filters disabled, which is only available to approved security teams—ordinary developers can't use it at all. On the email hidden-instruction test chart, Neon's attack success rate shows 0% versus Sol's roughly 2.4%, but small print notes that the 0% can only be achieved in conjunction with Google's additional security check layer and cannot be treated as the bare model being naturally immune.

Mervin ran five rounds of real-world tests using exactly the same prompts, giving each task only one chance. In the first round, building a space mini-game, both Neon and Sol ran successfully on the first try—the spaceship moved and there were no errors; Sol finished in under 40 seconds for only 5 cents, while Neon took 102 seconds and cost 20 cents—2.5 times the time and directly 4 times Sol's cost. In the second round, building a dynamic nighttime city scene, both failed completely: Neon ran the full 5 minutes without completing, burning 70 cents for nothing; Sol delivered a completely black screen. Re-tests in the X community confirmed the same pattern: building a dark church, Neon cost $2.67 versus Sol's $1.69; designing an electric toothbrush, Neon stopped midway while Sol completed it fully. The remaining three rounds compared Fable and an older version of Claude: on the bicycle model, Neon had the richest detail but took over 2 minutes—the slowest—at 24 cents; on a schedule program with 41 hidden tests, all three passed, with the older Claude costing only 9 cents, one-third of Neon's; on a precise math problem, Neon ground away for 37 seconds while the other two took only 10 seconds.

In the first round, building a space mini-game, both Neon and Soul ran successfully on the first try

Across five single-run real-world tests, Neon passed 4 challenges for a total cost of about $1.50—the capability is certainly there, but it's never the fastest, and never the cheapest.

Security filters and "agent version" models: Neon for Cyber in this article is an agent version with security filters disabled. Such versions typically remove or relax refusal policies (such as blocking exploit or malicious code requests) to serve legitimate security scenarios like penetration testing, but they are restricted to approved teams. The standard version used by ordinary developers has a security layer and will refuse or downgrade sensitive requests. Therefore, high scores in security tests cannot be directly applied to the public API—the public version is often actively restricted on offensive tasks, which is exactly the meaning of "0% attack success rate requires an additional security check layer."

How to Choose Without Paying the Stupidity Tax

Back to the original question: is Gemini 4 Neon really stronger than GPT-6.2 Sol? On the upper limits of complex programming and deep-thinking scores, Neon is indeed slightly ahead; but once you put speed, cost, single-run stability, and real tool conditions on the scales together, Sol's cost-effectiveness and response speed still hold a clear advantage.

If you're already subscribed to Gemini Pro or Ultra, it's a good deal to use Neon for complex coding once it rolls out; if you pay by API, keep using Sol or cheaper models for everyday small tasks and quick prototypes, and hand the hard nuts to Neon or the independently ranked #1 Claude Fable—that's the way to truly avoid paying the stupidity tax.

Benchmark reproducibility and vendor self-reported data bias: When evaluating AI models, the comparability of benchmark scores depends on three conditions: the same prompts, the same tool environment, and the same scoring scripts. Vendor launch events often mix third-party data, omit the strongest competitors, or configure advantageous tools for their own models, causing leaderboards to be distorted. The value of independent organizations (such as Valse and Artificial Analysis) lies in re-running under unified conditions, but even then, single-run variance remains large, requiring multiple samples averaged. Mervin's five single-run real-world tests in this article have a small sample size, so the conclusions should be treated as directional reference rather than statistical certainty.

#Gemini 4 Neon#GPT-6.2 Sol#Claude Fable#AI benchmark#cost-performance comparison
Share:

Related articles