Stop Shipping Skills Blindly! A Google DeepMind Engineer's Hands-On Guide to Agent Evaluation

Why Skills Can't Exist Without Evals

At a recent AI Engineer conference, Philip, an engineer from Google DeepMind, shared a sharp observation: nearly everyone is using Skills with coding agents, yet almost no one is writing evaluations (evals) for those Skills.

He ran a quick poll on the spot — virtually everyone raised their hand when asked if they used coding agents to write code, and a solid number used Skills too. But when it came to writing evals for those Skills, almost no hands went up. This is one of the most widely overlooked problems in AI Agent development today.

The most critical line he delivered: "Silent failures happen in production." These aren't the usual exceptions you'd see in conventional software engineering. The non-determinism of large language models stems from their sampling mechanism — even with identical inputs, a model may produce different outputs each time. When an agent task fails, it typically manifests as "output that looks plausible but is actually wrong" — no errors thrown, no trace left, nearly impossible to catch with standard monitoring. Because agents are inherently non-deterministic, when a task fails, it's hard to tell: was the Skill written poorly, or was the task simply too hard for the model? Without evals, you're developing in the dark.

According to data from SkillBench, which indexes over 50,000 Skills on GitHub, virtually none of them come with evals — and the majority were auto-generated by AI and never properly tested. The emergence of SkillBench itself reflects a structural gap in AI Agent development: the toolchain is expanding far faster than quality assurance practices can keep up. The flood of AI-generated Skills creates an "illusory reliability" — they look functionally complete on the surface, yet are riddled with hidden risks in production. This stands in stark contrast to the Test-Driven Development (TDD) movement in software engineering history: TDD took decades to become mainstream practice, and the culture of evaluating AI Agents is still in its infancy.

Using Agents vs. Building Agents: Two Completely Different Perspectives

Before diving into evaluation methods, Philip emphasized a core distinction that's easy to conflate: the agents we use and the agents we build are entirely different things.

The User's Perspective

When we use tools like Cursor, Claude Code, or Anti-gravity to write code ourselves, we're engineers with full context about the Skills involved. If the agent doesn't correctly invoke a Skill on the first try, we notice immediately, stop, re-prompt, or manually trigger it with a slash command.

The Builder's Perspective

But when you're building an agent inside an application for consumers or clients, the situation is completely different. Your users have no idea what a Skill is. They won't write "please use the refund Skill to solve my problem" in their prompts — they'll just describe what they need. This means that customer-facing agents must rely entirely on model-invoked Skills, with no option for users to trigger them manually.

This distinction determines the focus of evaluation: for agents built for customers, the quality of Skill descriptions is paramount, because the model must autonomously determine when to invoke them.

The Essence of a Skill: Progressive Disclosure

A Skill is essentially a folder containing a skills.md file, plus some supporting assets. Its core operating mechanism is Progressive Disclosure — a concept originally from UX design, where information is revealed incrementally based on user needs to avoid cognitive overload. In AI Agent Skills architecture, it's repurposed as a context management strategy, organized into three layers:

• Layer 1: Title and description. The description typically lives permanently in the model's context, letting the model know when to use the Skill.
• Layer 2: The Skill body, containing more detailed instructions and references to external files.
• Layer 3: Referenced files — the detailed context the model only dives into when needed.

This three-layer structure is essentially a lazy loading strategy: only when the model determines it needs deeper information does it trigger the reading of Layer 2 or Layer 3 content. Since a large language model's context window is a limited and expensive resource, injecting all information upfront causes costs to spike and attention to dilute. The three-layer structure is the core mechanism for balancing token cost against information completeness.

Philip also distinguished between two types of Skills:

Capability Skills: Teach the model to reliably do things it currently can't — like tracing logs or creating React apps. These are temporary — as models improve, they'll eventually be retired, and evals will tell you when a Skill is ready to be sunset.

Preference Skills: More durable, encoding team-specific workflows, style languages, or company-specific preferences. Base models typically won't incorporate this highly customized knowledge, making Preference Skills extremely valuable — and they must be protected by evals to prevent performance regression when the agent is updated.

According to SkillsBench 1.1 benchmark data, Skills improve performance by an average of ~15%, covering approximately 100 tasks across programming and productivity. The data also reveals an important conclusion: human-written Skills perform best, while AI-generated Skills can actually negatively impact performance — and skills.md files should be kept under 500 lines.

Eight Practical Tips for Writing High-Quality Skills

For model-invoked Skills, Philip offered eight concrete, actionable recommendations:

1. The Description Is Everything

Descriptions are often just two sentences, yet they determine whether the model triggers the Skill at the right moment. A weak description leads to over-triggering or failure to trigger when needed. Make sure the description covers three things: why (the reason to use it), how (the way to use it), and when (the right time to use it).

2. Write Instructions, Not Essays

Don't write descriptive prose like "The Interactions API is recommended for multi-turn conversations because it handles session state." Instead, give a direct instruction: "If you're building a chat application, use the Interactions API."

3. Stay Concise, Layer Your Information

Descriptions are a fixed cost paid on every model invocation (100–200 tokens), so they can't be long. For scenarios like multi-cloud deployments, put the specific instructions for AWS, Google Cloud, and Azure into referenced files — don't cram them into the main skills.md.

4. Set the Right Level of Autonomy

If a workflow is identical every single time (Step 1, Step 2, Step 3, fixed), you shouldn't use a Skill — write a script instead. Skills should define goals and constraints, not rigid step-by-step procedures.

5. Don't Ignore Negative Cases

We always focus on when a Skill should be used, but ignore when it shouldn't. If the description says "for web development tasks," both React and Angular projects will trigger it. Be more precise: "Only for React components or Tailwind CSS."

6. Test Early

When creating a new Skill, prepare 10–20 test prompts from the start: 5 "happy path" cases (scenarios where the Skill should be used), and 5 cases where it shouldn't be used, ensuring the model doesn't over-trigger. If you have real production data, include it.

7. Eliminate No-ops

Credit for this one goes to AI educator Matt: AI-generated Skills often contain a lot of no-ops — ineffective instructions that don't change agent behavior at all, like "make the implementation easy to read." The model already knows to write quality code — these instructions are pure token waste.

8. Know When to Retire a Skill

Skills don't live forever. Always compare eval results with and without the Skill loaded. If the model meets the bar without triggering the Skill, it's time to retire it and save on token costs and maintenance overhead.

A Real-World Case: The Gemini Interactions API Skill

Philip shared a complete, real-world example. The DeepMind team wanted to create a Skill for the Gemini Interactions API. Since this API was released after Gemini's last training run, the model had no knowledge of it and frequently generated code using the older Gemini 2.0 patterns.

The team built 117 test cases, sourced from real users generating Gemini code, synthetically generated test cases, and user feedback. Ultimately, performance on generating valid Interactions API code improved to nearly 90%.

The implementation was surprisingly simple — it only required two assets:

• A JSON file: Containing all test cases, structured with fields for prompt (user input), language (TypeScript/Python), should_trigger (whether the Skill should fire), and various expected checks (assertions).
• A basic Python script: Running the coding agent (Gemini CLI in this case), parsing the output and returning results.

Most assertions can be handled with regex — checking whether the correct SDK, correct model, correct method was used, and whether any deprecated patterns appeared. These assertions are extremely cheap to run and can be executed repeatedly. When a new model is released, you just update the model ID. For more complex Skills that require reviewing complete traces, you can introduce LLM as a Judge paired with a rubric to determine pass or fail.

LLM as a Judge has become one of the core paradigms in AI evaluation in recent years. The core idea: when task outputs can't be assessed for quality using rules or regex (e.g., code style, logical coherence, completeness), use a stronger language model as the judge. When paired with a rubric, the judge model scores each dimension according to pre-set criteria and outputs a pass/fail verdict. The advantage is flexibility — it handles open-ended outputs. The challenge is that the judge model may have its own biases (e.g., preferring longer answers, systematic preferences for certain styles), making high-quality rubric design critical. In Skill evaluation, LLM as a Judge typically complements regex assertions, specifically handling "complex trace judgments that simple rules can't cover."

This is how DeepMind manages Skills internally today: every Skill has an eval sitting right next to it, every change triggers an eval run, and a change is only allowed to merge if it improves the test pass rate — forming a strict regression testing mechanism.

Ten Best Practices and Ablation Testing

Philip wrapped up with ten best practices, calling out a few of the most commonly overlooked areas:

• Description is paramount: 50% of failures stem from Skills not being triggered correctly.
• Instructions over passive information: Tell the agent explicitly what to do, rather than describing background context.
• Include negative tests: This is the most commonly forgotten part.
• Start small: Even 10–20 eval samples is infinitely better than zero.
• Test outcomes, not paths: Don't test whether the model loads the Skill in the first turn — test whether it completes the task.
• Run in isolation: Coding agents are good at "cheating" — they'll pull context from conversation history to bypass the Skill.
• Run multiple trials: Due to non-determinism, each test case should be run 3–6 times.
• Test across harnesses: A Skill that performs well on Gemini may not perform well on Codex.

He also left some "homework": pick your most-used Skill and write 5 test prompts; try using a coding agent to analyze your own historical traces to identify your most-used Skills; review and remove no-ops to save costs; and most importantly, run Ablation Tests — always compare eval results with and without the Skill loaded. Only then can you know when to retire a Skill, or whether it's actually useful.

Ablation testing is a concept borrowed from neuroscience, where specific brain regions are removed to study their function. In machine learning, it's widely used to quantify the actual contribution of each component by removing model components one by one. In the Skill evaluation context, an ablation test works by holding all other variables constant and testing task completion rates under two conditions: "Skill loaded" vs. "Skill not loaded." This is the gold standard for determining whether a Skill is genuinely effective. If the base model achieves equivalent performance without the Skill, then the Skill isn't just unhelpful — it's burning extra tokens on every invocation. Ablation testing is also the only credible basis for "retiring a Skill" decisions, preventing teams from inertia-maintaining components that have long since lost their value.

The core conclusion comes down to one line: Don't ship skills without evals.

Key Takeaways