Agent Evaluation in Practice: Switching from Sonnet to DeepSeek for a 10x Cost Reduction

Introduction: A Model Switch That Cuts Costs by 10x

AI Agents running in production almost always default to the most powerful — and most expensive — models. But is that really necessary? In the third installment of this LLM evaluation series, the focus is on a core question: can a production Agent be downgraded from the expensive Claude Sonnet to a cheaper DeepSeek without sacrificing task quality?

The answer is an encouraging yes. Through a systematic evaluation process, the team successfully switched to the official DeepSeek V4 Flash release (0731). In the vast majority of scenarios, it performed on par with Sonnet 5, with only marginal degradation in a handful of edge cases — yet it delivered a more than 10x reduction in cost. This wasn't just a model swap; it demonstrated the enormous cost-saving potential that the engineering discipline of "evaluation" can unlock.

This article breaks down the complete methodology behind this evaluation, covering the business scenario, the five-component framework, key decisions in environment design, and the layered scorer architecture.

Business Scenario: A Deterministic CI Root Cause Analysis Agent

The subject of this evaluation is a production Agent whose job is to analyze the root cause of CI job failures in a code repository and decide whether a retry is warranted. This is a classic deterministic task — while Agents are typically general-purpose, in this scenario it only needs to do one thing well.

Two representative use cases illustrate its responsibilities:

• No retry needed: The Agent finds in the CI logs that a Jest snapshot mismatch caused the failure. Since neither the code nor the snapshot has changed, retrying would still fail — so no retry should be triggered.
• Retry needed: The Agent finds a connection reset network error when pulling a remote image, concludes the root cause is a transient network issue, and triggers a retry.

This kind of Agent significantly boosts developer productivity — engineers don't have to wade through error logs themselves, but instead see a direct attribution result, and retriable failures are most likely already handled automatically.

The Fundamental Difference Between Agent Evaluation and Prompt Evaluation: Side Effects

The core distinction from the previous "Prompt Evaluation" episode lies in side effects. When evaluating a single Prompt, the output is essentially text — even if it involves tool calls, as long as those calls aren't actually executed, there's no interaction with the outside world. But an Agent in a multi-turn execution actually triggers real tool calls, which produce real changes in the external environment.

In this scenario, there are two main categories of side effects:

1. Whether a CI Job is retried;
2. Whether a comment is posted on a Merge Request in GitLab.

These changes are generally not idempotent, so observing these changes and resetting the experimental environment becomes a unique challenge in Agent evaluation.

Managing side effects is already a classic challenge in software engineering — in AI Agent evaluation, it becomes even more complex. One of the core principles of traditional unit testing is "isolation" — each test should run from a clean initial state and leave no trace after completion. This is naturally satisfied for pure-text LLM calls; but for an Agent that makes real external API calls, each evaluation run may leave real comments on GitLab and trigger real CI retry jobs, operations that often can't be automatically reversed.

This means evaluation engineers need to design an additional "environment reset" mechanism: either clean up side effects after each test (delete comments, cancel jobs), or use an isolated test environment so that side effects don't affect production data. This is also the core motivation behind the design decision discussed later — switching from record-and-replay to a real GitLab instance — because only in a controlled real instance can side effects be safely produced, observed, and reset.

The Five-Component Framework: Dataset, Scorers, and Environment Design

Continuing the series' established "five-component" framework, this installment focuses on three areas: dataset, scorers, and the Agent-specific challenge of environment design.

Dataset: Distilling a Scenario Matrix from Real Sessions

Real data is far more valuable than synthetic data. The dataset for this evaluation came from the team's open-source NAP (Neutrae Agent Platform) project, where every Agent invocation generates a corresponding session record, naturally forming a real-world dataset.

From 2,893 raw sessions, the team used an Agent to filter down to 72 valid logs, which were then organized into four scenario groups:

• TA - Basic Decision: ~20 baseline decision samples;
• TB - Suppress Duplicates: When the same error appears repeatedly due to a new code push, the Agent should recognize it has already commented and the error hasn't changed, avoiding duplicate comments;
• TC - Real Anomaly Analysis: Errors that genuinely require a retry, requiring clear explanation to the user;
• TD - Fast Exit: If a patch is still in draft state and not ready, no analysis is needed even if there's a failure.

This yielded a final set of 38 carefully crafted scenario cases, including high-difficulty edge cases: jobs that were already manually retried, logs exceeding 4MB that GitLab's API can't return in full (requiring truncated queries), and more than 100 comments that require proper pagination to retrieve context.

It's worth emphasizing that much of this filtering and construction work was completed with the help of an Agent. The author repeatedly stresses a key point: once you start doing evaluation, you should lean even further into using Agents, because purely manual processing is simply too slow.

Environment Design: Pivoting from "Record & Replay" to a Real Instance

Environment design is the most critical aspect of Agent evaluation — it determines the environment in which the Agent actually operates. In this scenario, the GitLab API is the dominant dependency: the Agent needs it to query CI failure reasons, logs, current patch status, and more.

How to provide these APIs was the key question. The team considered two approaches:

1. Record & Replay: Low cost — record real API interactions for later playback;
2. Real GitLab Instance: Much higher cost — but the team ultimately switched from option one to option two.

This shift had deep underlying reasons. The core issue is that a real instance more accurately reflects the complex interactions found in production, avoiding the distortions that a recorded playback introduces during dynamic multi-turn calls. This serves as a reminder that the fidelity of the environment directly determines the credibility of the evaluation results.

Record & Replay is a common technique for testing API-dependent code. The principle is to intercept and save HTTP requests and responses in a real environment, then have subsequent tests read responses from the recorded files without actually making network requests. Typical tools include vcrpy in Python and nock in JavaScript. The advantages are speed, low cost, and no network dependency. The limitation is that it's a "static snapshot" — it captures a fixed request sequence at recording time. If the Agent takes a different call path during evaluation (e.g., querying comments before logs rather than the recorded order), the playback mismatches, causing test failures or returning incorrect data.

For a system like an Agent with dynamic decision-making capabilities, the call path itself is part of what's being evaluated — so the "path-binding" nature of record-and-replay severely undermines evaluation validity. Switching to a real GitLab instance increases maintenance cost, but preserves the Agent's freedom in actual calls, making evaluation results truly reflective of production behavior.

Layered Scorer Design: From Code Assertions to LLM Gold-Standard Review

The scorer design in this evaluation is quite sophisticated, consisting of three layers that balance deterministic verification with natural language quality assessment.

L1: Side Effect Assertions (Deterministic Code Judgment)

The bottom layer uses code directly to judge: was a comment posted? Was a Job retried? These are deterministic facts that code can evaluate completely — no model involvement required.

L2: Root Cause Analysis Quality (LLM-as-a-Judge + Gold Standard)

Comment content is natural language, and its quality and correctness cannot be assessed with deterministic code — so a model is needed as a judge (LLM-as-a-Judge). And for a model to judge, it needs a reference answer to compare against the actual output.

The key here is how the gold standard is built: the team used one of the strongest models available at the time (comparable to Claude Opus) to generate frontier-model reference answers for each session, then had humans review those answers, ultimately forming the gold standard.

This step is time-consuming due to human involvement, but the accuracy of the gold standard is paramount — if it's unreliable, the entire L2 analysis quality assessment loses its foundation.

LLM-as-a-Judge has become the mainstream approach for evaluating open-ended text outputs in recent years. The core idea is: when outputs can't be precisely measured by rules, use a sufficiently powerful model to simulate a human reviewer's judgment. This method has been validated in multiple academic papers as having high correlation with human ratings, but it also has known weaknesses — for example, "position bias" (favoring answers listed first), "verbosity bias" (favoring longer answers), and "self-preferencing" (models tend to rate outputs similar to their own style more highly).

Introducing a "gold standard reference answer" combined with human review is an engineering countermeasure against these biases: instead of scoring in a vacuum, the judge compares the candidate answer against a human-verified standard, converting the subjective question of "is this good?" into the more objective question of "how far does this deviate from the gold standard?" This makes evaluation results more stable, more auditable, and easier to reach consensus on within a team.

L3: Usage Statistics (Quantitative Analysis of Tokens and Latency)

This is a newly introduced layer that measures the number of Agent turns, Context Tokens consumed, and total execution time. The insight behind introducing this layer is quite profound:

In Agent tasks, a more expensive model doesn't necessarily mean worse cost-efficiency. A stronger model may use its higher intelligence to complete the task with fewer tokens and a shorter path; a weaker model, despite its lower per-token price, may end up costing more overall due to verbose paths and skyrocketing token usage.

So for Agents, evaluation is not just about "right or wrong" — it's also about efficiency.

Experimental Results: DeepSeek V4 Flash Wins on Cost-Efficiency

The experiment coincided with the release of the official DeepSeek V4 Flash (0731), while the Pro version was still in preview at the time. The Qwen 3.8 series had not yet been released, so Qwen 3.6's 27B and 35B MoE variants were used as alternatives.

The results closely matched DeepSeek's official benchmarks: on certain dimensions of this task, the official DeepSeek V4 Flash outperformed the Pro preview version, making it an even better value proposition. Compared to the original Sonnet 5, Flash performed slightly worse in only a handful of edge cases, but across the overall task it delivered a more than 10x reduction in cost — and then some.

The model evolution history is also worth noting: the Agent was originally built on Sonnet 4.6, and after approximately one week of tuning based on human feedback (primarily from developer users), it reached a satisfactory state. It was subsequently upgraded when Sonnet 5 was released, with no regression observed in practice and further improvements in phrasing and nuanced judgment.

Conclusion: Evaluation Is the Biggest Lever for Cost Optimization

The core takeaways from this practical exercise are:

• Downgrading Agent models for deterministic tasks is viable, but requires a systematic evaluation process to support it;
• Side effects and environment design are the key challenges that distinguish Agent evaluation from Prompt evaluation;
• Layered scorers (code assertions + LLM gold standard + usage statistics) cover both correctness and efficiency;
• Real data + Agent-assisted construction is the best practice for efficiently building evaluation sets.

The 10x cost reduction didn't come from luck — it came from a reusable evaluation methodology. For any team running expensive models in production, this may be one of the most worthwhile engineering directions to invest in.