Claude Sonnet 5 Hands-On Review: The Hidden Token Trap That Makes 'Low Cost' an Illusion

Article

Anthropic recently launched its biggest upgrade to the Sonnet series to date — Claude Sonnet 5. Officially positioned as "the most agentic Sonnet model ever," it promises significant improvements in reasoning, tool use, coding, and general knowledge tasks, with performance reportedly approaching the flagship Opus 4.8. Yet hands-on testing by content creators on both Bilibili and international platforms tells a different story: this may be one of Anthropic's most disappointing releases in recent memory.

Official Numbers: Performance Does Match Opus

On paper, Sonnet 5 looks impressive. According to official benchmark data cited by testers, Sonnet 5 achieves an average score of 63.2% across multiple evaluations. On Terminal Bench 2.1 it scores 80.4%, coming very close to Opus 4.8; on the HLE benchmark it nearly ties with Opus; its computer use score is 81.2%; and on the GDP EVOL benchmark it hits 16.19%, actually surpassing Opus.

These numbers confirm that Sonnet 5 has made real strides in agentic workflows. Agentic workflows refer to scenarios where an AI model doesn't just passively answer a single question, but actively breaks down complex tasks, calls external tools (such as browsers, code execution environments, APIs, etc.), and dynamically adjusts its action strategy based on intermediate results — forming a "perceive-plan-execute" feedback loop. Benchmarks like Terminal Bench and Computer Use are specifically designed to evaluate models' "hands-on" capabilities in such scenarios, simulating real automation tasks that require models to complete cross-system operations without human supervision. Sonnet 5 can plan tasks, invoke tools including browsers and terminals, and operate autonomously at a level that only larger, more expensive models could achieve just months ago.

Anthropic has also equipped it with a 1-million-token context window. This means the model can process roughly 750,000 English words in a single conversation — equivalent to several full-length novels — which is especially valuable for codebase analysis, long-document summarization, and complex multi-turn agent tasks. Early GPT-3 had a context window of only 2,048 tokens; a million-token window now represents the industry frontier.

Pricing also looks attractive at first glance. Through August 31, 2026, an introductory rate applies: $2 per million input tokens and $10 per million output tokens; after that it rises to $3 input and $15 output. On the surface, this looks like a highly cost-effective everyday workhorse model.

The Hidden Token Trap: Is "Low Cost" an Illusion?

The real problem, however, is buried in the fine print. Testers pointed out that Sonnet 5 has switched to the same tokenizer used in Opus 4.7, meaning the same piece of text is now split into approximately 1.1x to 1.3x more tokens than before, depending on content type.

Understanding this issue requires knowing how tokenizers work: a tokenizer is the core component that converts raw text into the numeric sequences (tokens) a language model can process. Common methods include Byte Pair Encoding (BPE) and SentencePiece. The granularity of a tokenizer directly determines how many tokens a given piece of text is split into — coarser granularity means fewer tokens; finer granularity means more. By switching Sonnet 5 to the Opus 4.7 tokenizer, Anthropic changed the vocabulary structure, causing the same input text to consume significantly more tokens, which directly raises the real-world cost for users.

In other words, while the unit price appears to have dropped, the same prompt now consumes more tokens — meaning actual all-in costs can far exceed expectations in certain scenarios. Testers pointedly noted that Anthropic's introductory pricing "is designed to keep costs roughly neutral after the tokenizer change" — a clever business move, but one that takes the shine off the word "cheap."

The problems become even more apparent in real-world benchmarks. CursorBench is a coding capability evaluation benchmark built by the team behind the AI programming tool Cursor. It focuses on measuring model performance in real software engineering tasks, including code completion, bug fixing, cross-file refactoring, and architectural design — widely regarded as an important reference for assessing a model's "engineering practicality." Sonnet 5 ranks only 13th on this leaderboard, a significant expectation gap for a series that has consistently marketed itself as a coding powerhouse. The Sonnet line's core value proposition has always been matching performance at a faster, cheaper price point than Opus. But now Sonnet 5's cost advantage has been dramatically compressed — it's only about 72% cheaper than Opus 4.8 Max.

This led testers to raise a sharp question: if you can get significantly stronger performance from Opus 4.8 for just a marginal cost increase, what's the point of using Sonnet 5 for everyday work? "Sonnet 5 may be the least token-efficient model ever, especially in max effort mode — it burns through tokens almost as fast as Opus while delivering weaker results."

Hands-On Testing: A Mixed Picture

To validate the model's real capabilities, testers ran Sonnet 5 through a multi-dimensional evaluation using a custom prompt library and the World of AI benchmark tool.

macOS Clone: A Surprising Bright Spot

In a macOS interface cloning task, Sonnet 5 performed surprisingly well. It generated a complete desktop environment with a login screen, notification system, switchable light/dark themes, and wallpaper-changing functionality. The top menu bar was mostly functional (with minor glitches), the bottom Dock was well-crafted, and app icons were drawn in SVG with a decent look.

Notably, apps including Launchpad, Safari, Messenger, Mail, Photos, Calendar, Notes, Maps, and Music were mostly functional. It even generated a playable FPS mini-game, a terminal, a calculator, and a text editor. The cost, however, was steep — the entire process took roughly 40 minutes in the workbench and consumed a massive number of tokens in max mode, making it extremely inefficient. It's worth noting that inference costs for very long contexts grow quadratically with window length (due to the complexity of the Transformer attention mechanism), which partly explains the unusually high token consumption for such complex tasks.

Minecraft and Frontend: Mediocre

In a Minecraft clone test, Sonnet 5 used a distinctive texture pack style, with working water physics and generated mobs including villagers and creepers. But issues abounded: no block-breaking animations, no inventory system, and no infinite terrain generation. The tester gave it 6.5/10 — the original textures were appreciated, but overall functionality was limited.

The frontend results were equally underwhelming. In a SaaS landing page generation task, while some components mimicked Anthropic's design style and scroll-triggered animations were present, the page was not fully functional, and a component in the top-left corner failed to render correctly. The tester bluntly noted that the open-source model GLM 5.2 produces better frontend output.

SVG Generation: The Most Glaring Weakness

If the earlier tests were a mixed bag, the SVG generation segment was a clear failure. SVG (Scalable Vector Graphics) generation is an extreme test of a language model's spatial reasoning — unlike generating text or code, SVG requires the model to precisely describe geometric shapes using coordinates, path curves (Bézier curves), color fills, and layer relationships in plain text, while building a corresponding visual mental model "in its head." This places extremely high demands on the model's spatial cognition, proportional perception, and aesthetic judgment. Current mainstream LLMs generally perform poorly on complex SVG generation tasks, partly because high-quality SVG code samples are relatively scarce in training data, and partly because models struggle to "foresee" the final visual result without executing the code. This is also why SVG generation is often used as a litmus test to distinguish a model's "genuine creative ability" from mere "surface fluency."

Testers asked the model to generate a side-view illustration of a BMW M4 CS. Even at the X-High setting, Sonnet 5's output was terrible — distorted proportions, awkward posture, and none of the brand recognition one would expect from a BMW rendering. Opus, by comparison, at least managed to sketch out the basic body structure and vehicle characteristics.

The tester expressed strong disappointment, concluding that for creative tasks, Sonnet 5's design sensibility doesn't even match GLM and should be ranked below GLM 5.2.

Verdict: Is Sonnet 5 Worth Switching To?

All things considered, the tester's verdict is blunt: Claude Sonnet 5 is "thoroughly disappointing." Beyond the agentic workflow improvements Anthropic touted, it doesn't deliver the leap many were hoping for. The supposedly lower price is significantly undermined by the new tokenizer — you burn more tokens while getting weaker results, making it hard to justify using Sonnet 5 over Opus 4.8 for almost any task.

That said, some perspective is warranted: this is a single reviewer's subjective evaluation based on custom benchmark tools, and Sonnet 5's official numbers for autonomous agent scenarios remain strong. Its real-world value still needs cross-validation from more independent evaluations. But this review at least highlights one important principle: don't just look at the unit price — look at the true all-in cost. The tokenizer change is an easy-to-overlook detail with far-reaching consequences — it quietly changes the actual bill behind every API call, and that kind of change rarely makes it into any headline.

The tester's final recommendation: for the vast majority of tasks, stick with Opus 4.8 to get the results you actually expect. And for Anthropic, this somewhat rushed release of Sonnet 5 may have only raised the outside world's anticipation for whatever comes next.

Key Takeaways