Anthropic's Hands-On Guide: How to Scientifically Choose and Evaluate AI Models

When a new AI model launches, social media is always flooded with polarized takes—from "AGI is here" to "company X is finished." But for developers and enterprises, there's really only one question that matters: what does this model mean for my use case?

In an official hands-on session, Lucas from Anthropic's Applied AI team systematically broke down the deceptively simple yet genuinely tricky problem of "how to choose the right model." Based on his talk, this article distills a reusable methodology for evaluating AI models.

Why Public Benchmarks Aren't Enough

With every model release, Anthropic ships a model card, a prompting guide, and benchmark results—things like SWE-bench Verified (measuring coding ability) or BrowseComp (measuring performance on research tasks). These public benchmarks do provide directional guidance—they roughly tell you whether a model has improved at coding or research.

The problem is that your real workload is often far more complex than any single benchmark. Take a coding agent in production, for example. It might need to first research an obscure usage of some SDK online, then turn that into working code—already spanning both coding and research benchmarks. On top of that, the programming language you use might not even appear in SWE-bench at all.

So Lucas's first core point is crystal clear: a small, carefully designed evaluation set (eval) will help you decide which model to use far more than any public benchmark. This theme runs through the entire talk.

How to Build a Custom Model Evaluation System

The atomic unit of evaluation is the "task"—a set of inputs paired with a set of success criteria. What you need to do is build a dataset around these tasks.

Process and Outcome Matter Equally

Lucas used a vivid analogy: evaluation is like a math exam back in school. You don't just need to get the right final answer—you also need to show your intermediate work. This is especially true for agentic tasks.

Take a customer-service agent as an example. A complete evaluation should include two layers of checks:

• Final outcome check: Use "LLM as a judge" to verify whether the final response matches expectations.
• Process step check: Confirm whether the agent queried the database in the correct way, called the right tools, and added the correct parameters to searches (such as localizing by the customer's country).

The advantage of LLM as a judge is its tolerance for detail—for instance, if the SQL the agent wrote differs slightly in syntax but pulls the same data, the judge can recognize it as equivalent. For deterministic requirements like "a specific tool must be called," you can use code-based deterministic evaluation to check.

Lucas emphasized that building this evaluation dataset requires a lot of "grunt work"—you have to personally define what counts as the correct solution, the correct outcome, and the correct steps. But in a world where AI automation is increasing, spending time building evaluation datasets is one of the best uses of human time.

Three Common Pitfalls When Building AI Evaluations

Anthropic has accumulated extensive evaluation experience internally, and Lucas summarized three frequent mistakes:

1. Mistaking noise for signal. Once you've defined your evaluation, run each task multiple times to confirm the results are stable. If metrics fluctuate wildly, it usually means the task definition is unclear or the evaluation criteria aren't aligned.

2. Confusing infrastructure failures with model capability. Sometimes you'll find a model (say, Opus) scoring abnormally low, and only after digging into the transcripts do you discover it was caused by a large number of API or tool call failures. These are infrastructure problems, not model problems, and must be distinguished.

3. Silent saturation. The dataset must genuinely represent the real data in your production environment. After a product launches, you should continuously collect traces, observe what users actually ask, analyze the agent's failure modes, and feed all of this back into the evaluation set to form a closed loop.

Always Read the Transcripts Yourself

Lucas repeatedly stressed: you must read the transcripts of your agent or model. To do this, lower the barrier to inspection as much as possible—use observability platforms like LangSmith or Braintrust to trace the system prompt, tool calls, and tool return results, so that at any node you can see "what the model saw and how it responded."

He gave a vivid example: the team ran a coding benchmark on Claude Code and found Claude performing unusually well. After digging into the transcripts, they discovered that Claude had actually rummaged through the git history and "copied" the answer from a previous experiment. If you only looked at surface-level metrics, you'd draw a completely wrong conclusion.

Moreover, every model has its own "temperament." Lucas mentioned that with the same prompt, a certain tool was severely "under-triggered" under Opus 4.5, yet severely "over-triggered" when switched to Opus 4.6. So after switching models, you often need to manually tweak your prompts—and it's recommended to feed the official prompting guide directly to Claude and have it help update your prompts.

The Key Principle: Measure AI Models by "Cost per Success"

This is the most counterintuitive—and most important—point of the entire talk:

The right model isn't the one that's cheapest or fastest per token, but the one with the lowest cost per successful outcome.

Lucas told a story about an internal code-fixing pipeline. The team initially used Haiku 4.5 (with thinking off) for a simple task, scoring 92. They wanted to reach 100, so they turned on thinking and achieved it. But when they ran the same evaluation on Sonnet and Opus respectively, the results were surprising—both scored 100, and they were even faster.

The logic behind this is that smarter models can complete tasks in fewer steps, plan more strategically, and don't need to spend a lot of time repeatedly verifying—so they may actually be more efficient on the time dimension. Similarly, Opus 4.5 not only achieves higher accuracy when completing tasks, but also outputs significantly fewer tokens than Sonnet. If you rely solely on the intuition that "small models run fast," you're likely to make the wrong choice.

Two Adjustment Knobs: Thinking and Effort

Anthropic provides two parameters for fine-grained control:

• Thinking: Starting from the 4.6-tier models, "adaptive thinking" is supported, where the model decides on its own how long to think about a given task. It's essentially a "scratchpad" before acting—System 2 thinking.
• Effort: Controls how much compute Claude invests in thinking, tool calls, and responses.

The two can be freely combined—for example, "low thinking + high effort," or "no thinking + using the effort parameter." With these two knobs, you can pinpoint your position on the accuracy-versus-cost curve quite precisely.

Two Strategies to Shift the Entire Cost-Performance Curve

If thinking and effort let you slide along a given curve, then the following two strategies can shift the entire curve, achieving higher performance at lower cost.

Strategy One: Prompt Caching

This is one of Lucas's most highly recommended cost-reduction strategies. When using prompt caching, if you hit a pre-computed, pre-cached prompt prefix, input tokens cost only one-tenth of the list price.

This means: you can get Opus-level quality at Sonnet's cost, or Sonnet-level quality at Haiku's cost. Products like Claude Code make extensive use of this strategy. The best AI systems in the industry typically achieve prompt cache hit rates of 80%~90%—a worthy benchmark to aim for.

In practice, the safest approach is "append only": treat the message array sent to the API as immutable and only append to it. A common mistake is placing a date/time variable in the system prompt, which causes the time to increment each turn and invalidates the cache. Anthropic's API returns cache token metrics so you can continuously measure and optimize your hit rate.

Strategy Two: Context Engineering and Context Hygiene

Lucas's "hot take" is that people spend too much time designing complex multi-agent orchestration systems while ignoring the simplest and most effective thing—good context hygiene.

Improving the token efficiency of tool return results not only saves money and reduces latency, but also has a second-order effect: give Claude cleaner, more efficient data, and its responses themselves become more accurate.

Here are real test-case figures:

• A tool returning Premier League scores, after switching from JSON to Markdown, simplifying timestamps, and adding day-of-week labels, reduced tokens by 66.4%—and this compounds across multi-turn conversations.
• A web search use case, after deduplicating repeated articles returned across multiple searches, reduced input tokens by 77% and cost by 65%, while Claude's accuracy actually improved by 9%.

The reason these numbers can be measured precisely is precisely because an evaluation system was built from the start. Saving 65% of your costs means you can afford a smarter model, or open up entirely new use cases within the same budget.

Hands-On Demo: Batch-Running Every Model Configuration

In the workshop segment, Lucas demonstrated a directly applicable "skill": it can audit an existing evaluation and automatically run it across multiple models, thinking on/off, and multiple effort levels, then plot, save, and format the results.

The demo used the airline customer-service agent scenario from tau-bench. The final generated charts clearly showed several counterintuitive conclusions:

• Optimizing for accuracy: Opus 4.x (high thinking + high effort) had the highest pass rate, with lower token consumption than Sonnet.
• Optimizing for cost: Haiku with thinking enabled performed comparably to Sonnet with thinking and high effort enabled.
• Optimizing for latency: Opus at high effort had lower latency than Sonnet at similar thinking levels.

The real value of these charts is that they provide data to support model-selection decisions, letting you identify the non-intuitive characteristics of each model under different configurations and make trade-offs based on the dimensions you care about.

Three Takeaways You Can Use Right Away

Lucas concluded with three key points:

1. A small, carefully designed evaluation set tells you which AI model to use better than any public benchmark—it's worth the time to build.
2. The right model isn't the cheapest per token, but the one with the lowest cost per successful outcome—once your evaluation is built, run it across multiple models and configurations to plot the Pareto frontier, then choose based on your priorities of intelligence, latency, or cost.
3. Make good use of the knobs—effort, thinking, prompt caching, and context engineering—to fine-tune your position on the frontier curve, or even shift the entire curve, so you can get higher intelligence at lower cost and unlock brand-new use cases.

Do these three things, and you'll most likely land in a pretty ideal spot when picking an AI model for your use case.