Gemini CLI Shutdown: Real-World Benchmark vs Claude Code — Which Is Faster and Cheaper?

On June 18, 2026, Google stopped offering Gemini CLI to personal Google accounts, migrating users to the new Antigravity CLI. This means past comparison articles claiming "Gemini CLI is the free first choice" are now outdated. On 2026-10-13, SideHub re-benchmarked Claude Code against Google's terminal agent using four types of real tasks, two runs per task, all in headless mode — with commands, prompts, and raw data fully public and reproducible.

Core Findings: Correctness Tied, Cost and Speed Worlds Apart

The benchmark used Claude Code 2.1.296 (claude-opus-5-5) and Antigravity CLI 1.3.4 (gemini-3.8-flash-high), default models, default effort, running on a Google AI Pro account. All 16 runs passed inspection, and every code review accurately identified the 3 planted bugs with no false positives. But the cost and speed gaps are enormous: Claude Code cost $0.169 per passing task versus $0.544 for Antigravity CLI — 3.2x cheaper; total time was 318 seconds vs 3712 seconds — nearly 12x faster.

You might not have noticed that Gemini 3.8 Flash's per-token price is actually about 5x cheaper than Opus 5.5, yet the default agy configuration used over 25x more input tokens and over 14x more output tokens to do the same work. This reveals a key lesson: per-token price is nearly useless for predicting per-task cost.

What Actually Happened to Gemini CLI

In a May 19 developer blog post, Google announced that Gemini CLI and the Gemini Code Assist IDE extension would stop serving Google AI Pro, Ultra, and free individual users on June 18. In testing, logging in with the previously working Google account failed outright:

Error authenticating: IneligibleTierError: This client is no longer supported
for Gemini Code Assist for individuals.

Gemini CLI itself hasn't been abandoned — it's still maintained as an Apache 2.0 open-source repository with ongoing model updates. But the replacement for individual users is the brand-new Antigravity CLI, installed as agy, written in Go, and not open source — its GitHub repo contains only a README, changelog, and examples, with no source code or license.

Antigravity CLI Migration Notes

• GEMINI.md and AGENTS.md are still read as usual
• Workspace skills migrated from .gemini/skills/ to .agents/skills/
• MCP servers moved from ~/.gemini/settings.json to ~/.gemini/config/mcp_config.json
• agy plugin import gemini can convert old extensions into plugins

Interestingly, Antigravity CLI doesn't only support Gemini: under a Google AI Pro account, agy models lists Gemini 3.8 Flash, Claude Opus 5.5, Claude Sonnet 5.5, and GPT-OSS 120B.

Test Methodology: Real Repo, Objective Checks

The test repo was python-humanize/humanize (MIT license, 1725 lines of Python, ~860 tests running in 7 seconds), pinned to a specific commit. The four tasks were:

1. Bug fix for a real issue — verified by upstream regression tests
2. New feature built to spec — 16 hidden acceptance tests
3. Behavior-preserving refactor — tests unchanged, old and new import paths return the same objects
4. Code review — 3 planted bugs; check whether all are found with no false positives

Each run happened in an isolated Docker container, mounting only the CLI binary and read-only login credentials — no home directory, no other credentials. Claude Code used claude -p, Antigravity CLI used agy -p, both with --dangerously-skip-permissions enabled.

ReDoS (Regular Expression Denial of Service) is the third type of bug planted in the code review test. Its principle: certain regex patterns, when matching specific forms of malicious input, have backtracking counts that grow exponentially, tying up the CPU for extended periods and creating a denial-of-service attack. For example, nested quantifiers like (a+)+$, when faced with a long string of a followed by a non-matching character, cause the engine to try every possible grouping combination.

Identifying ReDoS requires the reviewer to not only understand regex syntax but also reason about its complexity under worst-case input. In this test, Claude Code not only pointed out this risk but also measured the actual time cost to quantify the impact, whereas Antigravity CLI found the flaw but didn't verify it. This also explains why code review tasks demand far more "reasoning depth" from agents than ordinary bug fixes.

Detailed Results by Task

Bug Fix: 20x Speed Gap

Claude Code finished both runs within 30 seconds, with 4 to 6 tool calls; Antigravity CLI took 526 and 672 seconds, with 50 to 64 tool calls. The fix itself was one line of code, but agy spent enormous time on floating-point precision experiments, running mypy and ruff, and generating 38,000 thinking tokens. 93% of the total time was spent waiting for model responses, not executing tools.

Feature Implementation: Gap Narrows When Spec Is Clear

A precise spec left little room for implementation choices, and all four implementations passed the 16 hidden tests. Claude Code's second run used only 2 tool calls: one to read the module and tests, one to script the change and run tests.

Refactor: agy Slightly Better on Code Quality

This was the only task where agy performed better. The trap was that a helper shared by two modules would cause a circular import, and all four runs chose the same structure. The two agy runs additionally added a TYPE_CHECKING import and __dir__(), so both type checkers and autocomplete could recognize the new name. The cost: 10 minutes and 92 tool calls, versus Claude Code's 1 minute and 7 calls.

Code Review: Claude Code More Thorough

All reviews found the missing + 1, the integer truncation, and the ReDoS regex, with no false positives. The difference was in the details: Claude Code both times raised three additional valid points (wrong exception type for overly long input, GNU-style sizes rejected, localized output unparseable) and actually measured ReDoS timing. Antigravity CLI additionally found a precision-loss issue, but on its second run generated a 415-line uv.lock file when executing uv run ruff check . — despite the prompt explicitly saying "do not modify any files."

The Bottom Line: Tokens, Time, and Cost

Total across 8 runs	Claude Code	Antigravity CLI
Passed	8/8	8/8
Total time	318 seconds	3712 seconds
Input tokens	986K (91% from cache)	25.2M (93% from cache)
Uncached input	88	1.8M
Output tokens (incl. thinking)	23.1K (5.6K thinking)	332K (265K thinking)
Tool calls	37	437
Total cost	$1.35	$4.35
Cost per passing task	$0.169	$0.544

Is It the Model or the CLI?

To isolate variables, two additional single runs were done: dropping agy's model to gemini-3.8-flash-low still took 332 seconds and 67 tool calls; switching to claude-opus-5-5-high cut agy to just 51 seconds and 7 tool calls, close to Claude Code's 29 seconds. The slowness stems from how the default Gemini model works, not the CLI shell.

What is token caching? Every time a large language model's input is processed, it's billed — but when the same prefix (such as system prompts, repo files, conversation history) is sent repeatedly within a short window, providers cache it, and cache-hit input tokens typically cost only about 10% of uncached ones. This explains why the table's "input tokens" reach the tens of millions while "uncached input" is only 88 or 1.8M — most of the overhead comes from repeatedly read context.

But it also reveals a trap: cheap caching doesn't mean low total cost. If an agent resends a bloated context after every round of tool calls, even with 93% cache hits, the remaining 7% uncached portion can still be astronomical. In this test, Antigravity CLI's uncached input was 1.8M tokens — 20,000x that of Claude Code — which is the main source of the cost gap. So when evaluating agent costs, watch both "total input tokens" and "uncached ratio," not just per-token price.

Pricing and Quotas in Practice

Claude Pro and Google AI Pro are both in the $20 tier, both refresh every five hours with weekly caps, and neither publishes specific numbers. The key difference: Google explicitly states that Gemini Pro and Flash share a single quota, "deducted at API pricing." By that math, each agy task consumes about 3.2x the quota value of a Claude Code task. 100 similar tasks would cost about $17 with Claude Code and about $54 with the default agy configuration — rising to $109 once the Flash promotional pricing ends.

What is the "API-pricing deduction" quota mechanism? Google AI Pro and Claude Pro are both subscription-based, but they meter subscription allowances differently. Google explicitly states that Gemini Pro and Flash share the same quota pool, and that consumption is calculated at published API unit prices — meaning if a task consumes $0.544 worth of tokens at API list price, your subscription quota drops by the equivalent of $0.544.

Claude Pro's quota rules don't publish specific numbers, but from testing, the quota value consumed by Claude Code for the same workload is only about one-third of agy's. This means if you max out your quota each month, the same $20 subscription lets Claude Pro support roughly three times as many tasks as Google AI Pro. For heavy users, this gap directly determines whether you need to buy additional API credits, so when choosing, factor in both the "quota conversion rate" and the "subscription price."

Headless Runs and Permission Traps

All three support -p headless mode and JSON output. But there's an easy trap to fall into: without the skip-permissions flag, tools requiring approval are denied, yet the process still exits with code 0. In testing, both claude -p and agy -p returned 0 while doing nothing. CI scripts that only check exit codes would report empty runs as successful. You must check the permission_denials or denied_actions fields, or grant exactly the permissions needed.

Three-Way Comparison

On the same test set, Codex CLI (gpt-6.1-sol) cost just $0.074 per passing task — the cheapest; Claude Code was fastest with the deepest reviews; Antigravity CLI was slowest and most expensive under its default model. All three were comparable on correctness for these tasks.

Recommendations

When to prefer Claude Code: real-time tasks where someone is waiting (9-21x faster), code review (finds all planted bugs, with three times more valid findings, and doesn't pollute the repo), token-billed and scheduled tasks.

When to prefer Antigravity CLI: you already pay for Google AI Pro or Ultra and don't want a second subscription, overnight batch jobs with no one waiting, or you need multi-model options like agy --model claude-opus-5-5-high.

If you need an open-source, auditable agent: that's Gemini CLI with an API key, not Antigravity CLI.

A pragmatic approach is to split by task type: implement with Claude Code, do second-pass review with Antigravity CLI, and put shared rules in AGENTS.md, which both tools can read. SideHub's workflow lets you specify a provider (claude, codex, agy, copilot, or gemini) for each step, turning manual copy-paste into a traceable pipeline.

Limitations

Only two runs per task per CLI — a small sample; only one tidy small Python library, with the largest single request at 134K tokens, far from the million-token context window; Gemini CLI itself wasn't tested (personal login was rejected); only default settings; consumer account capacity is shared, so slower responses during busy periods can't be ruled out; the bug fix came from a public PR, so contamination is possible; pricing is as of 2026-10-13, and Gemini 3.8 Flash's promotional pricing will double on January 1, 2027.

The final honest conclusion: correctness is a tie, Claude Code clearly leads on speed and cost, and Antigravity CLI is worth using if you're already paying for Google AI Pro, your tasks aren't time-sensitive, or you pick a Claude model. Either way, re-run these four tasks on your own repo — the commands, prompts, and data are all public.

One-line Summary

After Gemini CLI's shutdown: Claude Code and Antigravity CLI tie on correctness, but Claude Code is 3.2x cheaper and 12x faster.

Paragraph Summary

Google stopped offering Gemini CLI to personal accounts on June 18, 2026, migrating users to the closed-source Antigravity CLI (agy). SideHub benchmarked Claude Code 2.1.296 (claude-opus-5-5) against Antigravity CLI 1.3.4 (gemini-3.8-flash-high) using four real tasks, two runs each, all headless. All 16 runs passed, and every code review correctly found 3 planted bugs with no false positives — correctness is a dead tie. But cost and speed diverge sharply: Claude Code cost $0.169 per passing task versus $0.544 for agy — 3.2x cheaper; total time was 318 vs 3712 seconds — nearly 12x. The root cause isn't per-token price — Flash is actually about 5x cheaper — but that the default agy config used over 25x more input tokens. Switching to claude-opus-5-5-high brought agy's speed close to Claude Code, showing that the slowness is how the default Gemini model works.