12 AI Agents Compared: Which Is Best Under DeepSeek V4.1?

A Hands-On Agent Test: 'From Solid to Slack'

When the same model (DeepSeek^Ad V4.1 Flash) is wrapped in 12 different Agent shells, how much can delivery capability vary? Bilibili creator "护身最高" (Hùshēn Zuìgāo) ran a comparison using 4 tasks close to real work, and the results were somewhat unexpected: the overall champion wasn't the most famous one, and the most expensive wasn't necessarily the most usable. This test cost 6.77 yuan, made 908 calls, and consumed 45.2 million tokens. In the end, one Agent scored a high of 96.5 on a single task, while another was ruled a direct loss for failing to deliver files.

Let's be clear first: this is a personal experience-based evaluation, not a comprehensive product review. All Agents used native prompts and native tools, each with its own independent API Key and independent working directory, with only the model unified. The same Agent may default to high thinking and long context; these are all recorded according to its own request parameters.

Test Design: 4 Tasks, Blind Evaluation, and Automated Acceptance

Tasks Come from Daily Deliverables, Not Drill Questions

The four tasks covered: cleaning a dirty sales spreadsheet and producing a briefing, fixing a discount bug in a real small repository, drawing "a teapot riding a bicycle" with code, and fixing a module configuration error. The creator emphasized that each task must leave behind files, test results, and reproducible instructions, with automated acceptance as a backstop—passing by altering tests to fool the pass rate does not count.

For example, in the bug-fixing task, hidden tests continue to check empty lists, discount boundaries, and CLI parameters; merely making the visible tests green is far from enough. In the configuration task, the source code is ESM but Node is told to parse it as CommonJS, producing the error "Cannot use import outside a module." The correct approach is to change the type field in package.json, not to delete the tests.

Scoring and Cost Accounting

In the composite score, quality accounts for 40%, speed for 20%, and cost for 40%. All tokens are uniformly calculated at off-peak prices, with the final amount based on actual deductions in the DeepSeek console. Blind evaluation was completed by the creator alone, with only one valid record per task.

Blind Evaluation is a practice in which the identity of the object being evaluated is hidden and scoring is based solely on output quality. In this article, the 12 Agents were anonymously numbered A to L (with one number left blank), and the creator first rated the delivery quality of each task on a 1-10 scale without knowing which product each number corresponded to, then revealed the numbers and combined them with cost and speed to calculate the composite score. The purpose of blind evaluation is to avoid brand bias or preconceived impressions affecting scoring, making quality scores closer to actual delivery level. Because only the creator scored and only one valid record was kept per task, this blind evaluation is closer to a personal experience-based assessment than to rigorous multi-person cross-review.

Automated Acceptance Testing refers to automatically determining whether a task is qualified through pre-written test scripts or checking programs, rather than relying on manual visual judgment. In the bug-fixing and configuration tasks in this article, hidden tests continue to check boundary conditions such as empty lists, discount boundaries, and CLI parameters; merely making visible tests green does not count as passing. The advantage of automated acceptance is that standards are uniform and reproducible, and it can effectively prevent cheating such as "altering tests to fool the pass rate," but it also requires the task designer to design test cases with sufficient coverage in advance.

Deliverables for the Four Tasks: The Details Behind Quality Scores

In the sales spreadsheet cleaning task, 239 of 300 orders were completed, with sales of 533,953 yuan and a refund rate of 14.3%. More noteworthy was the abnormal product area and next-step actions pointed out by the creator, rather than simply restating the numbers.

In the bug-fixing task, the discount parameter was originally a percentage, but the erroneous code directly subtracted it as a decimal; the correct approach is to divide by 100 first. The added category subtotal and JSON output must remain compatible with the original text output.

For the drawing task, the focus was on sitting posture, whether the feet are on the pedals, and whether the wheels are turning. In the configuration task, after fixing one place correctly, both tests and startup should recover; the final startup output should contain three records, a total of 240 yuan, and the highest category as advertising—not a single field can be missing.

Cost and Speed: Under the Same Model, the Shell Determines Wallet and Patience

The lowest cost for the four tasks was 0.35 yuan and the highest was 0.78 yuan, a 2.2x difference; the shortest time was 6.2 minutes and the longest was 24 minutes, a 3.9x difference. The creator specifically mentioned that T4Cloud Code took only 1 minute and Workbody spent 0.11 yuan, a 7.7x difference between the two—not because the model became dumber, but because of how much context the shell carried, how many rounds it sent back and forth, and how many words it made the model write.

A key money-saving fact: DeepSeek input is divided into cache hits and cache misses, and the price difference is large. In this round, the overall cache hit rate was as high as 97.6%, making input almost free. If cache is not counted, the estimate for the same tokens would rise from 6.77 yuan to 50.01 yuan; local recalculation based on peak/off-peak periods gives about 9.21 yuan. But the 1.2 million output tokens are the part that truly needs attention—some Agents like to resend the entire context again, and as the number of rounds increases, no matter how cheap input cache is, it will pile up.

Token Cache Hit and Cache Miss are key concepts in LLM API billing. When the input prefix of a request is exactly the same as that of a previous request, the server can reuse the cached computation results; this portion of input tokens is billed at the cache-hit price, usually far lower than the miss price; only newly added or changed input parts are billed at the miss price. In this article, the overall cache hit rate was as high as 97.6%, meaning that the vast majority of input tokens were settled at extremely low prices, and input cost was almost negligible. However, cache hits depend on stable reuse of request prefixes. If an Agent reorganizes or rewrites the entire context each round, the cache hit rate will decline and input cost will rise accordingly. Therefore, cache hit rate is not only a billing issue but also reflects an Agent's context management strategy.

Output Tokens refer to the amount of text actually generated and returned by the model, billed at the output price, usually several times more expensive than the input price. In this article, output totaled 1.2 million tokens, which is the part that truly needs to be watched. Some Agents tend to resend the entire context or generate lengthy intermediate steps, and as the number of rounds increases, output tokens accumulate quickly. Unlike input, output tokens cannot be amortized through caching, so controlling output length and reducing unnecessary back-and-forth rounds are key to lowering Agent usage costs.

Blind Evaluation Revealed: From "Completely Slacking" to "Solid"

The 12 Agents were anonymously numbered A to L (with one number left blank). The creator first gave blind scores from 1-10, then revealed the numbers. One vendor was ruled a direct loss because it failed to deliver files for the drawing task, and its speed and cost were zeroed out; Workbody delivered all four tasks but was the slowest at 24 minutes, and its score was dragged down by time and bill.

Final tiers: three in the "completely slacking" tier, five platforms in the NPC tier, two in the above-average tier, and two in the highest tier. The overall champion "Failure" scored 80.0 points, with a total cost of 0.35 yuan and total time of 6.2 minutes. The creator also recalculated once with quality first and once with budget first, and the first place did not change in either case.

Conclusion: The Model Sets the Ceiling, the Shell Determines the Experience

The core insight of this comparison is: the model determines the capability ceiling, but the shell determines how long you wait and how much you spend. If you care more about quality, raise the quality weight; if you only look at budget, raise the cost weight. The creator also reminds us that the results represent only these 4 tasks in this round; with a different task or version, the ranking could be completely different.

Configurations, questions, and per-run bills are all in the pinned reply; those interested can reproduce it.

Agent Harness/Scaffold refers to the software framework wrapped around an LLM that is responsible for organizing prompts, managing tool calls, maintaining context, controlling rounds, and handling error retries. When the same model is placed in different shells, actual performance can differ greatly: the shell determines how much context is given to the model, which tools it is allowed to call, whether it automatically retries, whether it compresses history, and so on. In this article, all 12 Agents used DeepSeek V4.1 Flash, but costs differed by 2.2x and speed by 3.9x; the root cause lies in differences in the shell's context management and round design. The shell does not change the model's inherent capability ceiling, but it profoundly affects the delivery experience—how long you wait, how much you spend, and whether files can be delivered reliably.

One-line Summary

12 Agents all using DeepSeek V4.1 Flash were tested on 4 tasks; the champion scored 80 and cost only 0.35 yuan.

Paragraph Summary

Bilibili creator "护身最高" placed the same model, DeepSeek V4.1 Flash, into 12 different Agent shells and ran a comparison using 4 tasks close to real work, covering dirty data cleaning, discount bug fixing, code drawing, and module configuration error fixing. The test used blind evaluation and automated acceptance, with quality accounting for 40%, speed for 20%, and cost for 40% of the composite score. The overall champion scored 80.0 points, cost 0.35 yuan, and took 6.2 minutes; under the same model, different Agents differed by 2.2x in cost and 3.9x in speed. An overall cache hit rate of 97.6% made input cost extremely low, but 1.2 million output tokens were the key to cost. The creator emphasizes that this is a personal experience-based evaluation: the model determines the capability ceiling, while the shell determines how long you wait and how much you spend.