Sakana Fugu vs GLM 5.2 Hands-On Test: Who Can Truly Challenge Top-Tier AI Models?

Another Model Arms Race

Over the past week, two new contenders emerged in the AI community, both claimed to challenge top-tier closed-source models: one is Fugu from Sakana AI, and the other is the Chinese open-source model GLM 5.2. A new "champion" pops up every week, and chasing the latest gets exhausting. This article is based on hands-on tests conducted by Bilibili creator Jack within the Hermes agent environment, helping you understand the real performance of these two models and how to choose between them in actual workflows.

It's worth clarifying up front: neither of these tests is a paper benchmark. Instead, they were placed into real Agent workflows (Hermes agent + Cloud Code), running head-to-head comparisons with unified prompts and isolated environments. This kind of "combat" evaluation reflects a model's usability in production far better than leaderboard-chasing.

On the testing environment background: Hermes is an AI agent framework designed for production workflows. Its core design philosophy is to decompose complex tasks into orchestratable skill modules (Skills), each of which can be independently bound to the optimal model. Cloud Code is a developer-oriented code execution environment released by Anthropic, supporting models to directly read/write files, run code, and invoke terminal commands—it is currently one of the mainstream solutions in the industry for agent operations on real codebases. Combining the two represents the most typical tech stack for deploying AI Agents today: the model handles planning and generation, the tool layer handles actual execution, and the framework layer handles state management and multi-step coordination. Under this architecture, a model's tool-calling stability, long-context comprehension, and output format compliance determine overall system reliability far more than single-turn Q&A accuracy.

Positioning Differences Between the Two Contenders

To understand this comparison, you first need to grasp the fundamental difference between the two models.

Fugu is essentially not a model, but an intelligent Router. When you converse with Fugu via API, hidden behind it is a pool of frontier models—Gemini, Opus, ChatGPT, MiniMax, and more. Fugu has been specially trained to automatically dispatch the most suitable model based on the type of question. Officially, it is defined as "a multi-agent system behind one API."

Intelligent routing is a meta-scheduling mechanism whose core idea is "let the most suitable model handle the tasks it excels at." From a technical architecture standpoint, the routing layer is usually composed of a lightweight classification model that analyzes the semantic features of the input request (such as code, mathematical reasoning, creative writing, tool calls, etc.) and then forwards the request to the optimal candidate in the backend model pool. This idea is not new—as early as 2023, projects like Martian and RouteLLM were exploring how to use small models to replace large models for "routing decisions" to reduce overall inference costs. The evolution of routing systems has gone through three stages: early reliance on manual rules (keyword matching, task-type enumeration), a middle stage evolving into semantic routing based on embedding vector similarity (making decisions by computing cosine similarity between the input and each model's "specialty description"), and more recently developing into end-to-end routing driven by lightweight discriminative models (such as BERT variants or distilled small LLMs)—these models are fine-tuned on large amounts of "task-type-to-optimal-model" labeled data and can make decisions with millisecond-level latency. However, as routing precision improves, the classification granularity becomes finer, causing the required volume of labeled data to grow exponentially, forming an inherent contradiction of "the higher the precision, the higher the maintenance cost."

Fugu represents the latest generation of solutions: deeply integrating routing with multi-agent orchestration, attempting to build an adaptive model federation behind a single API entry point. However, the benefits of a routing system are only worthwhile when they outweigh the extra round-trip latency and token overhead—and this hands-on test shows that Fugu has not yet reached this balance point on multiple tasks. For a routing system to truly deliver value, it requires both extremely low meta-scheduling cost and extremely high task classification precision; neither can be missing.

GLM 5.2's positioning is entirely different—it is an open-source, high-performance model dubbed the "open-source workhorse." Its biggest selling point is cost: it reportedly achieves results approaching or even surpassing Opus 4.8, yet at only one-sixth the price, while also supporting a million-level context window and local execution.

The GLM (General Language Model) series is jointly developed by Tsinghua University's KEG Lab and Zhipu AI. Its technical approach differs from the autoregressive architecture of the GPT series—early GLM used an Autoregressive Blank Infilling training objective, balancing bidirectional understanding and generation capabilities. This unique training objective gives the model stronger robustness when understanding incomplete contexts, but it also brings additional engineering costs for adapting to the mainstream ecosystem. Starting with the GLM-4 series, the model architecture converged toward the mainstream Transformer decoder and significantly improved code and math reasoning capabilities—this decision to converge toward mainstream architecture makes it easier to reuse inference optimizations accumulated by the community (such as quantization, speculative decoding, etc.) and lays the foundation for aligning tool-calling capabilities. GLM 5.2 supports a million-level context window (1M tokens), which is enabled by its RoPE position encoding extension techniques (such as YaRN or similar solutions): RoPE (Rotary Position Embedding) encodes positional information as the relative rotation angle between query and key vectors, theoretically providing stronger length extrapolation capability; improved schemes like YaRN further use temperature scaling and interpolation strategies to extrapolate the effective context length several times beyond the training range without retraining. Combined with memory optimization solutions like FlashAttention (which reduces the memory footprint of attention computation from O(n²) to near-linear), inference on ultra-long sequences becomes feasible on actual hardware. For Agent scenarios that need to process extremely long codebases or documents, ultra-long context is a key practical advantage rather than a mere marketing figure.

Test One: Tool-Calling Capability

The first round tests a model's ability to invoke tools—which is crucial for agents like Hermes yet often underestimated. The test task was to query the subject line of the latest email in an Outlook inbox via Zapier MCP. The three models (Fugu, GLM 5.2, Opus 4.8) executed the same prompt in their respective isolated environments.

On the tool-calling protocol background: Tool Calling (Function Calling) is one of the core capabilities of modern AI Agents. Its essence is to have the language model output, in a structured manner, "which tool I need to call and what parameters to pass," then have an external executor perform the actual operation and return the result to the model. This capability demands extremely high stability: the model must correctly parse the tool's JSON Schema definition, generate valid parameter formats, and handle errors and retries appropriately when the tool returns an error. Training Function Calling capability typically requires large amounts of "tool-calling demonstration data" for supervised fine-tuning (SFT), combined with reinforcement learning from human feedback (RLHF) or rule-based rewards (RLAIF) to improve format compliance. MCP (Model Context Protocol) is an open protocol standard released by Anthropic in late 2024, aimed at unifying the interface specifications between AI models and external tools/data sources. In terms of technical architecture, the MCP protocol uses JSON-RPC 2.0 as the underlying communication protocol and defines standard interfaces such as Tool Registration, Capability Declaration, and Streaming Response—this means that MCP-compliant tools (such as Zapier MCP) can be directly called by any model that has adapted an MCP client, without needing to write additional glue code. However, the protocol is still in a phase of rapid evolution, and its tool description format, error code specifications, and streaming response support differ across versions—this also partly explains why GLM 5.2 failed on its first call in this test.

The results were quite ironic:

• Fugu won this task, consuming about 230,000 tokens, with the lowest token consumption (even more economical than Opus 4.8).
• Opus 4.8 answered correctly, but consumed about 60% more tokens.
• GLM 5.2 failed—it failed to call the tool correctly on the first try and only succeeded on the third retry.

This result exposed GLM's obvious shortcoming: its strength is in coding, but its tool-calling stability is questionable. GLM 5.2's failure in the first round of testing likely stems from its immature adaptation to the specific Zapier MCP tool protocol—such protocol adaptation issues often do not reflect the model's underlying reasoning capability, but rather its integration maturity within a specific engineering ecosystem. However, in terms of latency, GLM was actually the fastest of the three (43 vs. Opus's 154). This shows that GLM is not lacking in capability, but is simply not yet stable enough in interfacing with Agent tool protocols.

It's worth noting that the cost of tool-calling failures is amplified in Agent workflows: a single failure not only consumes extra tokens (retries mean re-passing the full context) but may also interrupt the execution state of the entire task chain, preventing subsequent steps from picking up prior results. Therefore, the First-Call Success Rate of tool calls is one of the key metrics for measuring a model's reliability in production, rather than just whether the task is ultimately completed. From a systems engineering perspective, a model with a 95% first-call success rate versus one with 85% would differ by as many as 10 failures across 100 task executions—and each failed retry chain often consumes 2 to 3 times the tokens of the original task. This compounding effect can significantly impact cost and system stability in high-concurrency scenarios. The deeper risk is that task chains in Agent workflows usually have a directed acyclic graph (DAG) structure, where the failure of a single node blocks the execution of all its downstream nodes, causing the completion rate of the entire graph to drop in a nonlinear manner—meaning that in complex, multi-step, strongly-dependent tasks, the weight of tool-calling stability should be far higher than its apparent weight in single-turn testing.

Test Two: Frontend Website Generation

The second round required the three models to each create a single-page website for a sparkling water brand. The visuals needed to be attractive and include three page modules. The evaluation criterion was primarily aesthetics.

In terms of visual effect, GLM 5.2's output was far ahead—its bubble effects were the most outstanding, and it was rated the winner of this round. Fugu's work was "okay," but the bubble effects sank to the bottom, appearing somewhat contrived. Opus 4.8's version was rather basic, with oversized elements and disproportionate layout.

Even more noteworthy is the cost comparison:

• Fugu Ultra consumed 776,000 tokens, which was clearly excessive relative to its output. The wait time was 3 to 5 times longer than Opus 4.8, yet the quality improvement was quite limited. Internally, it made 15 API calls in total.
• GLM 5.2 had the lowest token count of the three, with speed roughly on par with Opus.

On token economics background: A token is the basic unit by which a language model processes text. Roughly, each English word corresponds to about 1.3 tokens, and each Chinese character corresponds to about 1.5 to 2 tokens (based on BPE or SentencePiece tokenization algorithms). In Agent workflows, each round of tool calling produces input tokens (historical context + tool definitions + tool return results) and output tokens (model response). The token cost of multi-turn conversations accumulates exponentially with conversation depth—a phenomenon known as "Context Inflation." Fugu making 15 internal API calls means its routing mechanism itself introduces substantial extra meta-information transmission overhead: each routing decision requires passing the task description, historical context, and routing metadata to the classification model simultaneously, then relaying the forwarded result back—essentially inserting an implicit "preparation for inference" before each actual inference. This "inference before inference" overhead produces a significant multiplier effect when task chains are long: if a single routing introduces 5% extra token overhead, 15 API calls mean a cumulative introduction of about 75% extra cost, which largely explains Fugu's abnormal consumption of 776,000 tokens. Using mainstream pricing as a reference, if input tokens are billed at $3/million and output at $15/million, a difference of hundreds of thousands of tokens per task can directly impact actual bills by hundreds or even thousands of dollars per month—which is decisive for the model selection decisions of high-frequency Agent application scenarios. For production systems executing thousands of tasks daily, token cost is often the primary operating expense besides GPU compute, and must be incorporated into the TCO (Total Cost of Ownership) framework during selection.

In other words, on this frontend generation task, GLM 5.2 not only delivered the best quality but also the lowest cost—clearly beating Opus 4.8 at the level of "initial style overview." And Fugu's routing mechanism exposed its core problem here: the extra latency and token overhead introduced often offset the benefits brought by intelligent dispatch.

Test Three: Real Codebase Improvement

The most difficult round handed the real code of the Hermes and Cloud Code agent operating systems to the three models, letting each improve the "memory tab" feature with completely open-ended approaches.

The completion time varied significantly:

• Opus 4.8: about 3 minutes
• GLM 5.2: about 6 to 7 minutes
• Fugu Ultra: 13 to 14 minutes

In terms of output quality, GLM's solution was the most outstanding—it designed a "memory health bar" presented with a green-to-yellow-gray gradient, balancing aesthetics and practicality. Opus made a recent-activity filter, while Fugu's output was highly similar to Opus's, yet took twice the time.

This once again confirms the core conclusion: Fugu essentially just routes multiple models together—the concept has value, but the latency cost it introduces is too high—the time to complete tasks is often twice that of Opus, while GLM's actual speed is even more competitive. In the real codebase task, GLM 5.2's million-level context window also showed its engineering value: being able to ingest a broader range of code context at once, reducing comprehension errors caused by context truncation, which is especially critical for code-improvement tasks.

On the engineering significance of long context and code comprehension: Traditionally, constrained by context windows (such as 4K, 8K, 128K tokens), models had to adopt strategies like "sliding windows" or "chunked summaries" when processing large codebases, which caused breaks in understanding cross-file dependencies. GLM 5.2's 1M-token context window can theoretically hold about 750,000 lines of code, enabling the model to simultaneously "see" a project's global architecture and local implementation details. In code-improvement tasks, this means the model can trace the complete path of a variable from definition, through passing, to final use, rather than "guessing" the context at a truncation point—this global awareness is often more decisive than single-point reasoning ability in large engineering projects. However, ultra-long context does not mean the model can uniformly "attend to" all content. Existing research (the "Lost in the Middle" phenomenon proposed by Nelson Liu et al. in 2023) systematically demonstrates that most Transformer models have a stronger attention bias toward the head of the context (the first few thousand tokens) and the tail (the last few thousand tokens), while the information retrieval accuracy of middle-section content declines significantly as context length increases. The root of this phenomenon lies in the position bias of the attention mechanism: models are more frequently exposed to shorter sequences during pretraining, so the learned distribution of their attention weights generalizes poorly to the middle positions of extremely long sequences. For code-improvement tasks, this means that if a key dependency definition happens to sit in the middle section of a million-token context, GLM 5.2's comprehension accuracy may be significantly lower than its performance on short sequences—this is the core reason why the actual engineering value of a 1M context still needs continued validation in larger-scale projects.

Who Really Won?

We must be honest: neither model truly "killed off" the top-tier closed-source models. In terms of overall capability, they have not yet comprehensively surpassed frontier models like Claude. But the real surprise of this test came from GLM 5.2.

Even if someone insists that Opus 4.8 is stronger, these rounds of testing offer little clear support for that. GLM 5.2 performed unexpectedly well, at only one-sixth the price. In terms of performance per unit token, GLM 5.2's cost-effectiveness is extremely outstanding. Its shortcomings are concentrated in tool-calling stability (especially the adaptation maturity for emerging protocols like MCP), rather than core reasoning or generation capabilities.

By contrast, Fugu's concept as an intelligent router is commendable, but in the hands-on test it did not become a "terminator"—the excessively high latency and token overhead greatly diminished the benefits routing brings. From an industry perspective, current mainstream routing solutions generally face a "dual dilemma of classification granularity versus scheduling cost": the finer the granularity, the higher the classification cost; the coarser the granularity, the more limited the scheduling benefit. Fugu's test results are a typical embodiment of this dilemma. From a deeper systems design angle, the true value of a routing mechanism only manifests in scenarios where task complexity is high enough and backend model differences are large enough—only when certain tasks genuinely require the unique capabilities of specific models (such as o3 for math reasoning, or specialized models for multilingual tasks) does the quality gain from routing become sufficient to cover the latency and cost overhead it introduces. For relatively homogeneous everyday tasks, the extra layer introduced by routing becomes a burden rather than an enabler. This conclusion also sounds an alarm for the product design of the entire routing space: the value proposition of a routing system essentially depends on the backend model pool being sufficiently heterogeneous—and as the capabilities of mainstream models converge toward homogeneity, the marginal benefit of routing will continue to decline.

How to Select Models Within Hermes

Selecting a model within a Hermes agent requires weighing four dimensions:

1. Performance: Quality of task completion
2. Speed: Response latency
3. Privacy: Whether it can be deployed and run locally
4. Cost: Expense per unit token

The recommended strategy is to turn Hermes into an intelligent "black box" that automatically routes to the most suitable model based on task type—which is precisely what Fugu theoretically aims to do but executes poorly.

Specific recommendations are as follows:

• Complex reasoning tasks: Prioritize GLM or Opus, and when necessary, let the two debate and reason dialectically with each other
• High-concurrency scenarios: Choose DeepSeek^Ad, whose performance approaches top-tier models but at an extremely low price
• Frequently used skill modules: Directly specify a dedicated model in the Hermes skill configuration, letting the system match automatically

For Agent developers, the truly efficient approach is to identify which few categories of common tasks account for 80% of your operations, then assign the model best suited to each category, rather than blindly chasing a single "benchmark champion." This saves both time and dramatically reduces costs. This "task-driven model combination" approach essentially implements the automatic routing goal Fugu pursues, but through lower-cost manual rules—at the current stage where routing technology is not yet mature, this is often the more pragmatic engineering choice. From a practical standpoint, it is advisable for developers to conduct task-type statistics on their own workflows in the early selection phase (such as the proportion of code generation, frequency of tool calls, proportion of creative generation, etc.), use this as a basis to build initial routing rules, then continuously optimize the model allocation strategy through A/B testing, ultimately forming a dynamic selection system with quantifiable ROI. It's worth noting that this manual routing system itself also requires version management: as each model's capabilities iterate rapidly, the optimal task-model mapping from last quarter may need recalibration after a new version is released—it is recommended to record model selection decisions as auditable configuration files (rather than hardcoded logic) to reduce the engineering cost of subsequent adjustments.

Conclusion

This "power-shift"-style model iteration teaches us that the value of a new model lies not in paper leaderboards, but in its actual performance within real workflows. Fugu's routing concept has promise, but its execution falls short; GLM 5.2, on the other hand, delivered a surprising answer at one-sixth the price.

For Agent developers pursuing cost-effectiveness, GLM 5.2 deserves serious inclusion in your model lineup, especially for frontend generation and code-improvement tasks, where it has full potential to replace expensive closed-source models. As tool protocol adaptations like MCP continue to improve, GLM 5.2's shortcomings in tool-calling stability are also likely to be fixed in future versions—at which point its overall competitiveness will further improve. From a more macro industry-trend perspective, GLM 5.2's performance also confirms that the capability gap between open-source and closed-source models is narrowing faster than expected: achieving performance close to top-tier closed-source models at one-sixth the price—if this price-performance relationship continues to evolve, it will have a profound impact on the entire commercial AI service pricing system. For the budget-sensitive but performance-demanding mid- and long-tail developer community, the practical value of open-source models is entering a new stage, moving from "usable" to "worth using."

Key Takeaways