[KongchangAI]
· 2 min read· 1,417 words

Gemini 4 Neon vs Claude Opus 5.5: When Benchmarks and Real Code Tell Opposite Stories

Gemini 4 Neon vs Claude Opus 5.5: When Benchmarks and Real Code Tell Opposite Stories

Benchmarks and real code tell opposite stories—toolchains skew results, so test models on your own workload.

Gemini 4 Neon wins 16 of 19 benchmarks, yet Opus beats it by 7 points on production code, and a 12-point gap appears in long-session debugging. The same Neon scores 39% in Google's AntiGravity but only about 13% in a neutral environment, showing how framework binding distorts evaluation. Neon matches or beats Opus on 3D and frontend work at far lower cost, while Opus leads on hardcore code and long text. Test on your own workload.

When Benchmarks Meet Real Code, the Tables Turn

Google claims Gemini 4 Neon wins 16 out of 19 benchmarks. But a code review company ran both models against its own production code—and Opus came out ahead by 7 percentage points. This isn't a simple scoring discrepancy; it reveals a long-overlooked dimension of model evaluation: how much the toolchain environment distorts real-world performance.

According to cross-validation from a YouTube channel and multiple developers on X, this conclusion is gaining traction among frontline users. But on tasks like 3D scenes and frontend prototypes, Neon's performance gives people pause. Who should you trust? The answer depends on what you're using it for.

Unlabeled allegory test

Day-One Testing: 3D Scenes and Gut Instincts

The earliest comparison to circulate was a 3D scene—single prompt, single HTML file. On this task, Neon clearly led Opus and even came close to Google's larger Carbon model. But other side-by-side comparisons went back and forth with no clear winner.

One developer who had used Opus heavily for three consecutive months said that after four hours with Neon, he realized he'd been "in denial." Another mixed outputs from Opus, Neon, and 4D together, stripped the labels, and asked people to guess which model produced which—the blind test was far harder than it looked.

But this doesn't mean Opus is out. The same tester gave both models identical prompts at maximum reasoning levels: Neon took 20 minutes and cost about $3.50; Opus took 18 minutes and cost about $7.20. Neon packed in more detail, but he still felt Opus's overall result was better.

Long Text and Style: Opus's Hidden Advantage

In a slide-generation test, Neon's first few pages looked good, but once the text got heavy, Opus pulled ahead—it preserved background effects while Neon went flat. Even more telling was the code volume difference: for the same spec Turbulon watch page, Neon wrote nearly 4,000 lines while Opus used only about 1,700—less than half—yet the tester preferred Opus's style.

The word "style" comes up repeatedly when discussing Opus. It's not a quantifiable benchmark score, but it's the key factor determining whether output is actually usable.

The code review company's production code test

Production Code Test: Opus Wins on Long Sessions

The most trustworthy test came from a code review company that put both models against its own production code in long debugging sessions, scored by independent reviewers. Opus scored about 87%, Neon about 80%. On long sessions the gap widened to 12 percentage points. More critically, there was stability: in one task, Neon made the same incorrect assertion about service behavior in three out of four runs—exactly the kind of path that ships bugs to production.

To be fair, Neon costs about 60% less per session and completes about 40% faster. But the lure of speed and cost can be offset by stability problems in long-chain debugging.

Long-horizon debugging is a critical but often oversimplified capability when evaluating coding models. It requires the model to maintain coherent memory of the codebase state, attempted solutions, and error history across dozens of interaction rounds while progressively narrowing down the problem. Unlike single-turn code generation, long sessions require the model to handle context window compression, accumulated tool-call errors, and dynamically shifting task goals. Opus's 12-point lead on long sessions shows it's stronger at maintaining long-term consistency—which typically relates to the model's context management strategy, its ability to detect errors in its own output, and the state-preservation mechanisms of the agent framework. Neon making the same incorrect assertion about service behavior in three of four runs exposes precisely the "stubbornness" problem in long sessions: once the model forms a wrong hypothesis, it struggles to self-correct through subsequent interaction. This failure mode is especially dangerous in real production environments, because developers may trust the model's assertions and skip verification, ultimately shipping bugs. Long-session stability therefore predicts a model's real-world usability for debugging work better than single-generation quality.

The Toolchain Trap: AntiGravity Holds It Back

How can Neon win 16 of 19 benchmarks yet lose here? A large part of the answer lies in the tool it runs in. On one coding benchmark, Neon scored 39% in Google's own coding tool AntiGravity, but only about 13% in a neutral environment. Same model, different environment, wildly different scores.

Google's chart also omitted Fable 5.2 and GPT 6.2 Astra, both of which scored higher. And AntiGravity is where complaints concentrate: some say it's unusable because the agent gets stuck on errors within minutes; others say it's better than before but still far from frontier tools.

Gemini plan users are stuck in AntiGravity

If you're on a Gemini Pro or Ultra plan, you're locked into AntiGravity and the Gemini app. Opus doesn't have this problem—even in Google's own benchmarks, Opus ran in Cloud Code, a tool many developers already use. Migrating Neon to other tools isn't smooth either: one person used Neon in Cursor and it repeatedly repeated the same step, while it worked fine in AntiGravity.

So when you choose between these two models, you're really choosing both a model and a tool.

When discussing model capabilities, one often-overlooked fact is that large language models don't directly execute code or manipulate files—they must go through an "agent harness" to call tools, read context, and execute commands. This framework breaks user tasks into steps the model can understand, manages multi-turn conversation state, and handles error retries. Different agent frameworks vary enormously in tool-call formats, context window management, and error recovery strategies, and the same model can perform vastly differently across frameworks. AntiGravity is the coding agent environment Google provides for the Gemini family, deeply integrated with Gemini models, but its tool-call stability and error-handling mechanisms have long drawn developer criticism. When a model is bound to a specific framework, benchmark scores no longer reflect pure model capability but the combined performance of "model + framework." This also explains why the same model scores 39% in AntiGravity but only about 13% in a neutral environment (such as direct API calls or third-party agents)—the former may have special optimizations or prompt adaptations for the framework, while the latter exposes the model's true level in general scenarios. Comparing models therefore requires specifying the evaluation environment, or the conclusions can be completely distorted.

Price and Willingness to Switch

Price is Neon's clear advantage. Per token, it's 25% cheaper than Opus. Per task, the gap is even bigger: on the Artificial Analysis Index, Neon's high-end version is 25% cheaper than Opus yet scores about the same as Opus's top configuration, at roughly one-third the cost and considerably less time.

Some people are indeed switching. One developer said Neon has finally replaced Opus as their daily driver; another said it's good enough to stop using Opus. One founder, after seeing early data, launched a production-environment blind test of Neon against Opus on day one—results pending.

Neon becomes the daily driver

Conclusion: Choose the Model by Workload

The consensus across multiple testers: for 3D scenes and frontend builds, Neon matches or beats Opus and costs far less; for hardcore production code and long text, people still lean toward Opus; many say they'd switch faster if Neon ran well outside AntiGravity.

One practical detail: keep Neon at its default high setting. The ultra-high setting adds only one point but costs about 70% more—and in one test actually scored lower.

Opus 5.5 has been out for two months, and Anthropic won't sit idle. The real advice: test it on your own workload and see whether it works for you. Benchmarks belong to others; your workflow is your own.

One-line Summary

Benchmarks and production-code evaluations diverge sharply, toolchain environments distort model performance, and choosing a model requires testing against your own workload.

Paragraph Summary

By comparing Google Gemini 4 Neon and Anthropic Opus in benchmarks versus production code, the article reveals how toolchain environments distort model evaluation. Google claims Neon wins 16 of 19 benchmarks, but in a code review company's long-session production code test, Opus won roughly 87% to 80%, with a 12-point gap on long sessions. The same Neon model scored 39% in Google's AntiGravity tool but only about 13% in a neutral environment, showing that framework binding heavily skews results. Neon matches Opus on 3D scenes and frontend builds at 25% to two-thirds lower cost, but Opus still leads on hardcore production code and long-text tasks. The article recommends testing on your own workload and notes that Neon's default high setting offers the best value.

#Gemini 4 Neon#Claude Opus 5.5#LLM benchmarks#AntiGravity#AI coding models#model evaluation
Share:

Related articles