Gemini 4 Carbon Hands-On: Clear Progress, But Still Not the Go-To for Coding

From Disappointment to Surprise: A Review Full of Twists

Google's Gemini 4 Carbon is finally available to developers, enterprises, and consumers. After extensive use, a YouTube tech reviewer with early access reached a contradictory conclusion: it's a massive improvement over its predecessor Argon, but still far from "good." The reviewer had originally prepared a "rant-style" video, but was swayed at the last minute by an unexpected performance—turning the review into an honest record of "the boundaries of model capability."

According to the reviewer, he also referenced third-party evaluation data such as Artificial Analysis. Gemini 4 Carbon briefly led in composite scores at launch, but was subsequently surpassed by other new models. This gap between "paper rankings" and real-world experience forms the core tension of the entire review.

Hallucination Rate Halved, But Still Not Reliable Enough

The metric the reviewer focused on most was the "false claim rate" in code analysis. In his Rust codebase architecture review test, Argon made 36 claims, of which 24 were false; Carbon made 79 claims, with 23 false claims.

Across all evaluations, the false claim ratio dropped from 36% for Argon to 18% for Carbon.

This data aligns with what the DeepMind team told him about "roughly a two-fold improvement." But the reviewer noted that an 18% hallucination rate is still far higher than other mainstream models (most have between 0 and 1 false claims, a few have 3 to 4). It fabricates database table column names, non-existent line numbers, and descriptions that don't match the actual architecture. For tasks like code review and codebase analysis, the reviewer explicitly says "absolutely do not use it."

False Claim Rate is a key metric for evaluating the reliability of large language models in code analysis tasks, referring to the proportion of claims made by the model in answers or reviews that don't match facts or the actual state of the codebase. This metric differs from general hallucination rate—it focuses on whether the model fabricates specific code structures, variable names, line numbers, or architecture descriptions. In code review scenarios, a high false claim rate means developers must spend significant time verifying the authenticity of model output, offsetting the efficiency gains from automation. The industry typically measures this ratio through manual annotation or automated comparison, using it to judge whether a model is suitable for direct use in production code analysis.

Speed and Cost: The Real Price Beyond Paper Specs

Carbon is marketed as a "fast model," but the reviewer's hands-on testing yielded the opposite conclusion. In Artificial Analysis testing, Carbon's average task time at the low reasoning tier was 22 minutes, and 27 minutes at the high tier; by comparison, Opus's low tier took only about 3 minutes. The reviewer emphasized that token generation speed (TPS) does not equal actual usable speed—time to first token and round-trip time are what determine the experience.

Pricing also has a "trap." Carbon is priced at $2 per million input tokens and $10 per million output tokens, but this is an "introductory price"—as of December 1, the price will double to $4 input and $20 output. The reviewer also found that the Gemini API's caching mechanism isn't transparent enough: within a 5-second window there's a 20% probability of a cache miss, 34% for a 30-second window, and higher for a 120-second window. Cache misses can increase request volume by up to 50%, making actual costs far exceed the list price.

Time to First Token (TTFT) and Round-Trip Time (RTT) are core metrics for measuring a model's actual interaction experience. TTFT refers to the time from sending a request to receiving the first output token, directly affecting the user-perceived response speed; RTT covers the total time for the request to travel through the network and be processed by the model. Unlike token generation speed (TPS), these two metrics are closer to real usage scenarios, especially in multi-turn conversations or coding tasks that require rapid iteration. High TTFT and RTT cause interaction lag—even if TPS is high, the user experience remains poor. Therefore, evaluating model speed requires comprehensive consideration of these end-to-end metrics.

Cybersecurity: The Underrated True Highlight

Midway through the review, the reviewer admitted he had "slept on" the cybersecurity section. Google emphasized Carbon's frontier performance in cybersecurity capabilities at launch and has opened it to security defenders through the Fair Wind program.

According to Google, Carbon achieved a 25% relative improvement over Argon in Project Fortify and discovered two defects in popular open-source projects with severity comparable to PHP remote code execution vulnerabilities. In prompt injection tests, Carbon achieved a 0% injection success rate; its frequency of cheating in evaluations also dropped significantly from Argon's 4%.

The reviewer was initially skeptical of these claims, but handling a real security report changed his mind. An independent developer discovered two security issues in T3 Code through Anthropic's security program. The reviewer first had other models handle it, but the results were either refusals or impractical solutions (such as requiring users to manually generate a six-digit verification code on another machine). Carbon, after understanding the problem, directly proposed a solution to "hard-isolate the known machine flow from the raw URL flow" and additionally discovered two derivative issues not mentioned in the report.

Ultimately, the fix received a 9-point rating from Opus 5.5 and Muse. The reviewer admitted: "I didn't expect the best experience handling these reports to come from this model."

Prompt Injection is a class of attacks in the LLM security domain where attackers embed malicious instructions in inputs to induce the model to deviate from its original task or leak sensitive information. In cybersecurity evaluations, prompt injection success rate is an important metric for measuring model robustness—a 0% success rate means the model can effectively identify and reject such attacks. Project Fortify is a Google initiative aimed at using AI to discover security vulnerabilities in open-source software and drive fixes through collaboration with the security community. Carbon's performance in this program reflects its capabilities in real vulnerability discovery and remediation recommendations.

Engineering Collaboration and Ecosystem: DeepMind's Responsiveness Deserves Recognition

The reviewer used Gemini through the T3 Code ACP binding. In early versions, the ACP binding had data transmission issues that prevented the model from getting key context such as line numbers—which was also a source of some hallucinations.

After receiving feedback, the DeepMind team audited the ACP implementation and completed fixes, and also added special identifiers to T3 Code to provide a better experience. The reviewer praised this highly: "If I ranked labs by willingness to fix problems, I'd put DeepMind first." However, he also noted that Gemini CLI is no longer open source, contrasting with earlier strategy.

ACP Binding (Agent Communication Protocol Binding) is a protocol implementation for integrating Gemini with external tools (such as T3 Code), used to pass context information (such as line numbers, file structures, etc.) between the model and the development environment. When ACP binding has data transmission issues, the model cannot obtain complete context, causing generated content to diverge from actual conditions and exacerbating hallucinations. DeepMind's rapid fixes for such integration issues reflect its emphasis on the developer ecosystem and also affect the model's usability in real engineering scenarios.

Should You Use It? Conclusions by Scenario

Scenarios where you should NOT use Carbon:

• Code review and codebase analysis (hallucination rate still too high)
• 3D content generation (hands-on results clearly fall short of expectations)
• Interactive programming requiring fast responses
• Large codebase comprehension (the reviewer believes it struggles with larger-scale projects)

Scenarios worth considering:

• Cybersecurity incident response and vulnerability remediation design
• Existing users within the Gemini ecosystem who need an upgrade
• Exploratory tasks where you're willing to "babysit" the model and guide it step by step

The reviewer's final judgment: Carbon is "a huge step" over Argon, with noticeably smoother daily use and better task focus retention. But it's still not his default choice for daily programming—Opus remains ahead in cost-effectiveness, response speed, and engineering quality.

"I wouldn't say Google 'cooked' successfully this time, but it's definitely less bad," the reviewer wrote at the end. He's considering moving Carbon out of the "Jester" tier in his evaluation system to observe its long-term performance. For users locked into the Gemini ecosystem, this is a worthwhile upgrade; for developers with options, it's not yet the first choice.

One-line Summary

Gemini 4 Carbon halves hallucination rate but code review remains unreliable; cybersecurity shines, speed and cost have traps.

Paragraph Summary

A tech reviewer with early access conducted an in-depth evaluation of Gemini 4 Carbon with contradictory conclusions. Carbon's false claim rate in code analysis dropped from Argon's 36% to 18%, but remains far higher than mainstream models—unusable for code review. Real-world speed is far slower than advertised: average 22 minutes at low reasoning tier versus Opus's ~3 minutes; pricing is introductory and doubles December 1, with opaque caching driving actual costs higher. However, Carbon excels in cybersecurity: 0% prompt injection success rate, dramatically reduced cheating rate, and high-quality solutions in real vulnerability fixes, earning 9-point ratings from Opus 5.5 and Muse. DeepMind responded actively to developer feedback, quickly fixing ACP binding issues. The reviewer ultimately concludes Carbon is a massive improvement over its predecessor but not the default choice for daily programming—worth upgrading for those locked into the Gemini ecosystem.