Claude Sonnet 5 In-Depth Testing: 8 Real-World Tasks Reveal Its True Capability Boundaries

Introduction: Sonnet 5 Is Here, But It's Not That Simple

Anthropic has officially released Claude Sonnet 5, positioned as "performance approaching Opus 4.8 at a lower price." It sounds appealing, but what's the real-world experience actually like? Bilibili creator KaterSony conducted an in-depth test of Sonnet 5 across 8 real-world task scenarios (covering image recognition, 3D modeling, web page generation, procedural scenes, and more).

This article is based on that testing, mapping out Sonnet 5's true capability boundaries, as well as its strengths and weaknesses compared to competitors like GPT-5.5, Gemini 3.1 Pro, and Composer 2.5. Here's the key takeaway upfront: Given enough tokens and time, Sonnet 5 produces remarkably high-quality output, but the "cheap" selling point is actually questionable.

Benchmarks and Pricing: Impressive Performance, But the Cost Math Is Tricky

Based on officially released data, Sonnet 5's performance shows a distinct pattern of "uneven strengths":

• On SWE-Bench Pro (a software engineering benchmark), Sonnet 5 clearly lags behind Opus 4.8;
• On the Knowledge Work dimension, however, it actually surpasses Opus 4.8.

SWE-Bench was introduced by Princeton University in 2023, with a design philosophy of pushing model evaluation from "writing syntactically correct code" to "solving real engineering problems." The original SWE-Bench pulled 2,294 real issues and corresponding fix commits from GitHub, requiring models to generate valid patches based solely on issue descriptions and codebase context—without knowing the answers. The Pro version further raised task difficulty by introducing cases with more complex cross-file dependencies and stricter test coverage, making it harder for models that rely purely on memorized training data to fake their way through. It's considered one of the gold standards for measuring an AI coding assistant's "true engineering capabilities." The Knowledge Work dimension, on the other hand, focuses on office scenarios like information retrieval, document analysis, and multi-step reasoning. The two evaluate significantly different capability directions, which explains why Sonnet 5 exhibits this "uneven strengths" phenomenon.

Anthropic officially emphasizes that Sonnet 5 performs strongly in browser operations and terminal use. In the AA benchmark, Sonnet 5 Max's intelligence level ranks just below Opus 4.7 Max, but its cost to complete a single task is higher than the latter's.

It's worth noting that the "AA benchmark" (Artificial Analysis Intelligence Index) is a comprehensive AI model evaluation system maintained by the third-party organization Artificial Analysis, which factors in a model's intelligence level, response speed, and API cost. Current mainstream AI evaluation systems have fundamental philosophical disagreements in their design: Chatbot Arena (LMSYS) uses a blind-test ELO scoring system, relying on human conversational preference votes—its results are close to real user experience but heavily influenced by audience taste; the SWE-Bench series pursues "quantifiable verification of real tasks," using unit test pass rates as the sole standard; the Artificial Analysis Intelligence Index weights intelligence level, inference speed, and API pricing across three dimensions, targeting API developer scenarios. This fragmentation of evaluation systems reflects the industry's lack of consensus on the very definition of "AI intelligence"—labs also tend to choose benchmarks favorable to them when promoting their models, further exacerbating information noise. Similar systems include Cognition's Frontier Code (focused on code agents), among others. Different benchmarks use varying evaluation methods, datasets, and weightings, resulting in significant discrepancies in the same model's ranking across different benchmarks—which is also the deeper reason behind the "contradictory benchmark conclusions" discussed below, reflecting the current reality that the AI evaluation field has yet to establish a unified standard.

There's also a subtle pricing trap worth mentioning: Sonnet 5 has switched to a new Tokenizer, consuming more tokens to process the same content—officially stated to be 1.0 to 1.35 times that of Sonnet 4.6.

A Tokenizer is a core component of large language models that converts raw text into numerical sequences, responsible for splitting continuous character streams into discrete token units. Mainstream approaches include Byte Pair Encoding (BPE), WordPiece, and the Unigram Language Model. BPE works as follows: it first splits text into individual characters, then repeatedly merges the highest-frequency character pairs found in the training corpus through statistics, ultimately forming a vocabulary that balances common and rare words. Different tokenization strategies show significant compression rate differences for content like Chinese, code, and mathematical formulas: Under BPE systems, Chinese typically consumes 1-2 tokens per character, while English words can often be encoded as a single token; structural characters in code such as indentation and brackets show even more dramatic consumption differences across tokenizers. OpenAI's tiktoken, Google's SentencePiece, and Anthropic's proprietary tokenizer differ significantly in compression rates for Chinese, code, and special symbols. After Sonnet 5 switched to the new tokenizer, token consumption for Chinese and mixed content clearly increased (officially reported as 1.0 to 1.35 times the original), which directly amplifies costs in API billing scenarios. For Chinese-speaking users or scenarios requiring processing of large volumes of Chinese documents, the practical impact of this change is particularly pronounced. Developers should reassess their budget models before migration and empirically test the actual token inflation rate in their own use cases.

This means that even at a lower listed price, the actual usage cost—after token amplification—may not truly be cheaper. Promotional discounts are, in a sense, precisely meant to offset this change.

In horizontal comparisons, Composer 2.5's intelligence on Knowledge Bench surpasses Sonnet 5 Max, with far lower per-task costs; GLM 5.2 was rated as "very cost-effective." On Cognition's Frontier Code benchmark, Sonnet 5's score even exceeded that of Opus 4.8. The contradictory benchmark conclusions precisely demonstrate that—no single model can dominate across all dimensions.

Image Recognition: Dominant Performance in Max Mode

The first test asked Sonnet 5 to identify multiple bugs hidden in an image. The creator selected the website's highest "Max" thinking level.

Sonnet 5 spent about 10 minutes and adopted a rather "brute-force" strategy: repeatedly zooming the image to extreme detail, then using code to identify each position one by one. It ultimately output both a "locally zoomed version with bug positions annotated" and a "full-image annotated version," and the recognition results were completely correct.

For comparison, Gemini's top-tier model—previously considered the strongest at image recognition—found only 4 bugs in the same image; GPT-5.5 X-High failed to detect them entirely.

This case reveals Sonnet 5's core trait: It doesn't rely purely on visual understanding, but excels at "invoking tools + multi-round iteration" to converge on the correct answer. The combination of zooming, coding, and comparison enables it to surpass traditional vision powerhouses on tasks requiring extreme detail. This strategy essentially transforms a "direct perception" problem into an "algorithmic search" problem—reducing the difficulty of each judgment by exhaustively examining local regions, trading time for accuracy.

Notably, this "tool-augmented perception" pattern closely mirrors how human visual experts work: When identifying camouflaged species in the field, entomologists likewise use magnifying glasses to scan region by region, rather than relying on holistic impressions. This reveals a general principle of current multimodal models: On fine-grained perception tasks, a "tool invocation + structured search" strategy is often more reliable than single-pass end-to-end reasoning. From a technical perspective, Sonnet 5 here effectively decouples the visual problem into two sub-problems—first, obtaining high-resolution local images through code cropping (bypassing the detail loss from original image compression), then independently performing object detection reasoning on each local region. This "divide and conquer" strategy can theoretically transform any low-confidence full-image detection problem into multiple high-confidence local detection problems, thereby substantially improving recall.

3D Modeling and Procedural Scenes: The Extreme of a Detail Obsessive

The second group of tests better showcases Sonnet 5's "engineer personality."

SCAD Format 3D Modeling

The creator had Sonnet 5 generate a .scad file that could be opened in OpenSCAD (which allows generating 3D models via code for 3D printing).

OpenSCAD was born in 2009, with a design philosophy of letting engineers precisely describe geometry using declarative scripts rather than manually manipulating objects in 3D space. Unlike visual modeling software such as Blender or Fusion 360, it describes geometry entirely through code—its core syntax includes primitive geometries (cube, sphere, cylinder), Boolean operations (union, difference, intersection), and transformations (translate, rotate, scale). Boolean operations are the soul of OpenSCAD: difference() can "carve out" one shape from a solid to create grooves or holes, while intersection() keeps only the overlapping part of two shapes. Combining these two operations can describe almost any complex mechanical part. This "code-as-model" paradigm is naturally suited to parametric design (adjusting dimensions just by modifying variables), and it naturally fits how large language models generate—the model only needs to output valid .scad scripts rather than directly manipulate 3D space. The "check environment → code → render → view image → iterate" workflow that Sonnet 5 demonstrated in this task essentially incorporates visual feedback into the code correction loop—bridging the gap between "syntactically correct code" and "visual effect meeting standards," reflecting its synergy between tool invocation and multimodal reasoning.

The web version of Claude comes with a built-in virtual machine, so Sonnet 5 first checks whether OpenSCAD is installed and whether it can print, then after coding continuously renders results, views images, and iterates for improvement. Although extremely time-consuming, the final product's quality was quite good, clearly superior to the mediocre standards of GPT-5.5 and Composer 2.5.

Suzhou Museum Three.js Scene

This task best illustrates Sonnet 5's "obsessiveness." Using Sonnet 5 Max paired with Claude's official Harness, it spent over an hour and still hadn't finished, with an extremely high number of tool invocations, thinking mostly in English, and even checking whether the environment had a headless browser and invoking Playwright to run extensive tests.

Three.js is built on top of WebGL, which itself is a browser implementation of OpenGL ES, allowing JavaScript to directly invoke the GPU for real-time rendering without installing any plugins—widely used in fields like data visualization and product display. The Suzhou Museum scene is so challenging because it simultaneously involves multiple high-difficulty rendering techniques: PBR (Physically Based Rendering) requires material parameters (roughness, metalness, normalMap) to precisely match real-world lighting models, with its core being the Cook-Torrance microfacet reflection equation, which can simulate how light scatters on rough microscopic surfaces; water surface reflections require real-time reflection mapping techniques, typically achieved by using CubeCamera to dynamically render the surrounding environment and map the result onto the water material; the alpha transparency blending of plant textures is highly prone to Z-Fighting sorting errors when multiple layers overlap, requiring precise control of rendering order. These technical difficulties require the model to simultaneously possess Three.js API knowledge and graphics fundamentals.

Playwright is Microsoft's open-source browser automation testing framework, supporting operation in "headless mode"—meaning it doesn't launch a graphical interface but only executes browser operations in the background and takes screenshots via page.screenshot(). After the AI generates code, Playwright renders the page in an isolated environment and takes screenshots, which are then fed back as visual input to the multimodal module for quality judgment, forming an automatically iterable closed loop—this is precisely the underlying mechanism that enables Sonnet 5 to continuously optimize scene details, and it also makes visual verification steps beyond code syntax correctness possible. This "generate-render-visual judgment-correct" closed-loop workflow holds significant paradigmatic meaning in the field of AI-assisted frontend development: It automates the "visual acceptance" step that traditionally required human intervention, theoretically allowing infinite iteration until visual quality meets standards.

But the result was quite stunning: The ground material, the texture of the pebble path, the reflections of the bonsai, and the texture of the rockery stone slabs were all the best in quality among all the models the creator tested. The fixed version even added a body of water in the middle, making it more aesthetically pleasing overall. Even minor details like how pine needle textures render under simple lighting were checked and rendered one by one.

The conclusion is clear: Sonnet 5 Max pursues ultimate quality, at the cost of enormous time and token consumption.

Web Page and UI Generation: Solid, But Not Omnipotent

In web page generation tasks, Sonnet 5's High mode performed excellently:

• Headphone website: The texture and design of the ear cups were done well, with a style close to the prompt's "Apple aesthetic" requirement—the overall vibe was on point;
• Desktop orrery: Also in High mode, but the performance was "quite mediocre," indicating fluctuations in its stability across different tasks.

In the 31-page AI recipe app prototype task, a key division-of-labor pattern was exposed: GPT-5.5 did the best at deciding what content to put on each complex page, while Sonnet 5 is better suited for the subsequent interface generation. This division-of-labor pattern reflects an essential difference between the two types of models: GPT-5.5's advantage in "content planning" may stem from its training depth on structured content like product documentation and user research reports; while Sonnet 5's advantage in "interface implementation" is more reflected in precise control over CSS layout systems, component hierarchies, and visual consistency.

From an engineering practice perspective, this capability distribution difference provides important reference for multi-model collaborative workflows: In complex product prototype design scenarios, "content architecture planning" (information architecture, user journeys, page function definitions) can be handled by GPT-5.5, then the structured page specifications can be fed as context into Sonnet 5, which specializes in precise implementation at the visual layer. This "planning and execution" division-of-labor model also hints at a possible paradigm for future multi-model collaborative workflows: Different models each perform their own roles, chained into a pipeline rather than a single model doing everything. Compared to the version generated by Opus 4.8, the two aren't very different (both being Claude family), but Opus 4.8 additionally organized "10 section categories," which was more thorough.

The Shortcoming of the Foldable Phone Stand

When completing the "OPPO foldable phone multifunctional stand" task in Cursor, Sonnet 5 processed for over 20 minutes, taking screenshots from multiple angles to inspect, but the final product still had obvious flaws—the phone was placed too close to the rear support plate. In contrast, Gemini 3.1 Pro completed it in just two or three minutes in AI Dev, with well-designed anti-slip silicone strips, storage slots, and a charging port design—higher overall completion and higher efficiency.

This comparison reveals the fundamental trade-off between "deep iteration" and "rapid output": The quality gains from Sonnet 5's prolonged thinking don't necessarily cover its time cost across all task types. For engineering design tasks with clear physical constraints (such as a stand's ergonomic spacing and structural stability), the model needs to simultaneously consider both the abstract "design sense" and the concrete "dimensional constraints"—and the latter often needs to be explicitly provided in the prompt rather than left for the model to infer on its own. This also suggests an optimization strategy: In such tasks, providing detailed constraint specifications upfront (e.g., "the gap between the phone and support plate should be no less than 5mm") is often more efficient than relying on the model's iterative self-correction.

Traffic Simulation and the "Submarine" Failure Case

The final two tasks likewise had their hits and misses.

Intersection traffic simulation (High mode, processed for 25 minutes in Cursor): The vehicle designs were aesthetically pleasing, but the traffic light placement had issues. The creator deliberately provided richer and more accurate prompts for this type of task, and the effect did improve—which also validates the earlier experience of testing with Haiku: Giving the model detailed prompts significantly improves the results. However, the relationship between pedestrians and vehicles was never handled well.

The submarine scene was a clear failure: The submarine wasn't visible from the default view and had to be seen by rotating; when the ship moved forward, the submarine should have tilted forward but didn't; other scene elements were also done quite simply.

These two cases collectively point to a deeper problem: Physical simulation involves domain knowledge like rigid body dynamics and fluid mechanics approximations, while existing large language models' "physical intuition" essentially derives from textual descriptions of physical phenomena in the training corpus, rather than actual physics engine simulations—this fundamentally differs from how professional physics engines (such as Bullet Physics, PhysX) work by numerically integrating Newton's equations. The pitch attitude of a submarine moving forward in water involves complex force relationships such as propeller thrust and the relative positions of the center of buoyancy and center of gravity. This kind of knowledge often appears in the corpus in qualitative descriptions rather than quantitative formulas, making it difficult for the model to precisely implement at the code level. The handling of pedestrian-vehicle relationships in traffic simulation is similar: Real traffic flow simulation relies on professional algorithms like cellular automata or social force models (such as the widely-used Helbing social force model). The coverage density of such precise mathematical models in general training corpora is far lower than that of routine programming knowledge, leading the model to tend toward replacing real physical interaction logic with simplified geometric position judgments. This fundamental limitation is fully exposed in complex scenes, and no amount of iteration can compensate for the missing understanding of underlying physical logic.

Conclusion: Give It Enough Resources and It's Powerful, But Don't Expect Cheap and Fast

Across all 8 tasks, Sonnet 5's capability profile is now clear:

Strong scenarios: Fine-grained image recognition, 3D modeling, procedural visualization scenes (like the Suzhou Museum Three.js), and web page generation with clear style requirements—as long as it's given enough tokens and iteration time, the final product quality is often the best in its class.

Weaknesses and costs:

1. Thinking process is too long. Even in High mode, whose thinking volume is far less than Max, Sonnet 5's reasoning process remains verbose and largely in English.
2. Cost is questionable. The new tokenizer increases token consumption by 1.0 to 1.35 times, and "Sonnet 5 is not necessarily cheaper than Opus 4.7."
3. Not recommended for lightweight tasks. For simple needs, neither the time nor the cost is worth it.
4. Not omnipotent. For scenarios like rapid prototyping, content planning (GPT-5.5 is stronger), and efficiency-first (Gemini 3.1 Pro is faster), it's not the optimal choice.

In one sentence: Sonnet 5 is a "detail obsessive" pursuing ultimate quality, suited for complex, high-quality-demanding, heavy-duty tasks that can tolerate long iteration times—not for everyday lightweight use.

From a broader perspective, Sonnet 5's capability characteristics reflect a deep trend in current AI model development: As model scale and reasoning capabilities improve, the strategic space for "trading time for quality" is expanding—given a sufficient computational budget, models can approach human expert-level output quality by extending reasoning chains and increasing rounds of tool invocation. The cost of this capability is nonlinear growth in inference costs, which also places higher demands on the user side for "task tiering": Identifying which tasks are worth investing in high-cost deep reasoning and which can be quickly completed with lightweight models will become an increasingly critical decision dimension in AI toolchain design.