Eight AI Models Tested: A Full Record of Dory's Production Environment Model Selection

When an AI Agent enters production, choosing which underlying model to use is no longer a leaderboard game — it becomes a three-way trade-off between accuracy, response speed, and cost. The Playtomic team recently conducted a rare "real-world" evaluation: they put eight new models into the actual operating environment of the company's internal AI Agent Dory, using real questions from real employees to test the models' true capabilities.

What Dory Is: An AI Colleague Living in Slack

Dory is Playtomic's company-wide AI Agent, permanently stationed in Slack. Engineers use it the most, but the main source of questions comes from non-technical teams such as customer service, product, and marketing. It works more like a colleague: querying Looker for data, reading GitHub code, checking Linear tickets and Notion documents, pulling Datadog logs, and then replying in the conversation thread.

Dory handles about 1,000 conversations per month, and a single answer may involve dozens of steps. Questions fall mainly into three categories: data-related (about two-thirds, such as "how many clubs used feature X last month"), product behavior-related (answered by reading code, such as "why didn't this match show up in the app"), and knowledge and operations-related (how processes work, who is responsible for a certain step, how to change configurations).

At the time of this test, Dory was running on OpenAI's GPT-5.6 Terra Pro.

Why Test It Yourself Instead of Looking at Leaderboards

Public leaderboards measure general capabilities: math problems, self-contained programming, common-sense Q&A. But the tasks Dory faces are completely different — it has to understand a hastily written Slack message, figure out where to find the answer, chain together multiple tools, understand Playtomic's own data and code, and finally give a short but correct reply.

A model can rank at the top of a leaderboard yet perform mediocrely on this kind of task. So Playtomic's approach was: use real questions and a real Agent, changing only the underlying model. The eight models tested included:

US camp: OpenAI's Terra Pro (the current model) and GPT-6 Luna Pro; Anthropic's Claude Sonnet 5.5 and Claude Haiku 5.5; Google's Gemini 3.8 Flash.
China camp (strong price competitiveness): GLM 5.3 and DeepSeek^Ad V4 Pro.
European representative: Mistral Large 4, but because it did not yet support Zero Data Retention (ZDR) — Playtomic requires vendors not to retain any data sent to them — it was only tested for reference this time.

What are public leaderboards? Such as LMArena, SWE-bench, MMLU, etc., these are common industry evaluation leaderboards for model capabilities. They usually use fixed question sets (math, programming, common-sense Q&A, etc.) and score models in a controlled environment, with the goal of providing a "general ruler." The advantage of such leaderboards is that they are reproducible and horizontally comparable, but their flaws are also obvious: most questions are self-contained, require no external tools, do not depend on a company's private data, and almost never involve multi-step tool orchestration. Yet the performance of an AI Agent in production is precisely determined by these "not tested by leaderboards" capabilities — whether it can understand vague natural-language requests, whether it can correctly select and chain multiple tools, and whether it can understand the company's own data definitions. The reason Playtomic abandoned leaderboards and built its own evaluation is precisely this mismatch.

Testing Method: Using Real Questions to Test a Real Agent

Several key decisions in the test design are worth noting.

Authenticity and lack of bias in the question samples. The team randomly sampled real conversations, preserved a mix of normal question types, and removed personal information. One detail in particular: they did not keep only positively rated answers — because that would bias results toward the current Terra model. In the end, 39 questions were selected.

Running in production. Each model ran inside the production version of Dory, with identical tools and instructions, and could only read, not modify, anything. Because Playtomic's data changes every day, all models answered the same question back-to-back to ensure they faced the same data.

Rules first. Before seeing any answers, the team had already defined the standard for "as good as the current model."

AI judge mechanism. Two models from different companies served as judges: Anthropic's Claude Opus 5.5 and OpenAI's GPT-5.6 Sol. Neither could see the other's judgments. The judges did not simply "read" answers; they "verified" them: first deriving the correct answer from code, documents, and tickets, then for data-related questions checking item by item whether the query statement, metric definitions, and cited numbers matched what the query returned, and finally rating each answer as correct, partially correct, incorrect, unsupported, or unanswered. Length, tone, and confidence were not scored.

To eliminate bias, the team adopted several measures: answers were anonymized as A, B, C and randomly ordered; a second round of letter reshuffling was conducted for review, and the two rounds were basically consistent; the two judges were averaged to offset each one's favoritism toward its own model; a control test was set up in which a third judge was secretly fed an identical pair of answers, and all results were judged as ties; manual spot checks were conducted where judges disagreed, and the judges' judgments were each time equal to or better than the manual checks. GPT-5.6 Sol was the stricter judge, but the two judges' rankings of the models were completely consistent.

Zero Data Retention (ZDR) stands for Zero Data Retention, a data compliance agreement between an enterprise and a model vendor: the vendor promises not to store or use for training any data sent by the customer (including prompts, tool-call parameters, and returned results). For a company like Playtomic that sends internal code, tickets, and user data to third parties through an Agent, ZDR is a hard admission threshold, especially when GDPR and other compliance requirements are involved. Because Mistral Large 4 did not offer ZDR at the time, it had to be excluded from the formal ranking and was only tested for reference — a reminder to selection teams: no matter how strong a model's capabilities are, if it cannot pass compliance, it cannot enter the production candidate pool.

Results: The Triple Trade-off of Quality, Speed, and Cost

Quality: How Often Answers Are Correct

A correct answer scored 1 point, a partially correct answer scored 0.5, and the two judges' scores were averaged and converted to a 100-point scale.

Model	Quality Score
Claude Sonnet 5.5	74
Gemini 3.8 Flash	70
Mistral Large 4	70
GPT-6 Luna Pro	61
Claude Haiku 5.5	59
GPT-5.6 Terra Pro (current)	56
GLM 5.3	56
DeepSeek V4 Pro	45

Speed: Every Minute Matters in the Slack Scenario

Users wait for Dory's reply during their work, and a good answer within one minute is better than a perfect answer within fifteen minutes. The team measured typical latency and the slowest 10% latency.

Claude Sonnet and Haiku were the fastest and most stable, never exceeding 3 minutes. Mistral and Gemini took more than 10 minutes on about one out of every four questions, with some approaching 25 minutes. This is the main reason Mistral, despite matching the best in quality, was temporarily out of the running.

Cost: The Actual Cost per Answer

Cost is less important than quality and speed, but at the scale of 1,000 conversations per month it cannot be ignored. The key insight is: price per token does not equal price per answer. Gemini and Mistral have cheap per-token prices, but their average number of steps per question is as high as 38, making the cost per answer comparable to or even higher than the current model. Caching also has a huge impact, reducing costs to one-third to one-sixth, and the caching effect varies significantly across models, so public token prices alone have limited reference value. The costs below all include caching.

Model	Vendor	Quality	Cost per Answer	Typical Latency	Slowest 10%
Claude Sonnet 5.5	Anthropic (US)	74	$0.35	51 sec	2.1 min
Gemini 3.8 Flash	Google (US)	70	$0.79	3.3 min	16 min
Mistral Large 4 (no ZDR)	Mistral (EU)	70	$0.57	3.5 min	17.5 min
GPT-6 Luna Pro	OpenAI (US)	61	$0.05	2.3 min	6.7 min
Claude Haiku 5.5	Anthropic (US)	59	$0.03	44 sec	1.4 min
GPT-5.6 Terra Pro (current)	OpenAI (US)	56	$0.60	1.9 min	4.4 min
GLM 5.3	Zhipu (China)	56	$0.27	1.1 min	5.1 min
DeepSeek V4 Pro	DeepSeek (China)	45	$0.23	1.3 min	4.0 min

Why can caching significantly reduce costs? Large model APIs usually bill input tokens and output tokens separately. Each time an Agent answers, it must send a large context to the model, including system instructions, tool definitions, and conversation history. If this were fully recalculated at full price every time, the cost would be very high. Prompt caching allows vendors to cache repeated context fragments, and subsequent cache hits are billed at a very low discounted price (commonly 10%–25% of the original price). Because an Agent's instructions and tool descriptions are highly repetitive within the same session, the cache hit rate is often very high, so the actual cost can drop to one-third to one-sixth of the "no-cache theoretical value." This also explains why the article emphasizes: different models have significantly different caching effects, and looking only at the public per-token price is of limited significance; you must look at the cost per answer after including caching.

Where Each Model Falls Short

No model is perfect in every dimension, and understanding shortcomings is more valuable than looking at total scores.

Claude Sonnet 5.5: Occasionally asks for clarification instead of answering directly, but apart from that has the highest accuracy.
Claude Haiku 5.5: Gives partial answers to ambiguous questions and has weaker coding ability.
GPT-5.6 Terra Pro (current): Asks for clarification when it does not know the asker's identity; sometimes trusts outdated documentation instead of code.
GPT-6 Luna Pro: When it does not know the asker's identity, it asks for clarification on more than one-third of questions; some answers are extremely slow.
Gemini 3.8 Flash: Gets stuck in repeated-step loops; on two questions it gave up after more than 200 steps.
Mistral Large 4: Extremely long investigation process, up to 269 steps; does not yet support ZDR.
GLM 5.3: Weak coding ability; sometimes confuses which entity in the system is performing an action.
DeepSeek V4 Pro: Claims certain things "do not exist" without verification and exaggerates numbers; also attempted to perform write operations that no one requested.

The "step count" and "loop" problem of Agents deserves separate explanation. An Agent is not like an ordinary chat model that only generates a piece of text; it must repeatedly iterate in a loop of "think — call tool — read result — think again" until it gives a final answer. Each step is a complete model call, consuming tokens and time. Therefore step count directly determines cost and latency: the more steps, the more expensive and slower it is. An even worse situation is getting stuck in a "loop" — the model repeatedly performs the same action, repeats queries, or keeps trying without progress, eventually giving up after exhausting the step limit (the Gemini case in the article of giving up after more than 200 steps is exactly this problem). Step count reflects both the model's planning ability for tool use and its judgment of "when to stop," and is an indicator in Agent selection that is closer to real experience than pure quality scores.

Methodological Limitations That Must Be Honestly Acknowledged

This was a practical test for Dory selection, not a general benchmark, and the team also candidly listed its limitations:

Each question was run only once. Models do not give the same answer twice, and rerunning the same question on the same model can cause score fluctuations of up to 10 points.
Only 39 questions. Enough to trust the model ranking, but not enough to trust precise gaps. Product questions (11 questions) and process questions (2 questions) were even thinner.
The judges are also models. Although averaging across two companies can reduce self-preference, it cannot eliminate it completely.
Default settings. Including the amount of reasoning before each model answered. Other settings could change quality and cost.
Mistral was run only once. ZDR was disabled only for that test.
Time snapshot. Vendors update models frequently, and these data are from October 2026.

Conclusion: Who Is More Suitable in Different Scenarios

Claude Sonnet 5.5: Both judges ranked it first, typical response time was under one minute, cost per answer was about 1.7 times cheaper than the current model, and monthly cost was about $350.

Claude Haiku 5.5: Monthly cost about $25, quality close to the current model, and the fastest speed. Suitable for simple tasks, or as Dory's internal second-tier model.

Mistral Large 4: Top-tier quality, but too slow for the Slack scenario and lacking ZDR; if both aspects improve, the team would test it again. Chinese models currently do not meet the bar for this use case, mainly losing on accuracy.

These results come from Playtomic's own questions, data, tools, and priorities (such as fast answers in Slack), and may not hold for another company or another use case. But this evaluation method of "real questions + real Agent + dual-judge verification" has reference value for any team currently doing Agent model selection: public leaderboards can only answer "is the model smart?", while production needs to answer "is the model reliable in my scenario?"