Eval-Driven Development: How to Scientifically Test Your AI Skills and Agents

Why AI Tools Need Evals

In episode 27 of Firebase After Hours, Google senior engineer JF — with 13 years at Google, recently focused on Evals and AI — made a core point: if you're building AI tools, writing Agent skills, or setting up MCP servers, you need to actually test them rigorously. Not just run them five times in a terminal and say "looks good."

By testing, he means applying the same rigorous methods as traditional software engineering: the mindset behind unit tests and end-to-end integration tests applies equally to AI integrations. The key difference is that LLMs are inherently non-deterministic. In traditional software, 2+2 always equals 4 — results are reproducible. Ask an LLM "why is the sky blue," and you might get 14 differently worded answers that are all correct. How to score this kind of "uncertain" output is the core problem Evals are designed to solve.

From String Matching to "LLM as a Judge"

JF breaks testing methods down into several levels.

Prioritize Deterministic Testing

If a task can be verified deterministically — say, having an Agent generate code — use traditional methods: Does it compile? Does it pass the linter? Does it pass pre-written unit tests? This is the gold standard: fast and reliable. For scenarios like iOS development, you can even use grep to check key strings — for example, verifying whether code uses the new @Observable or the deprecated @Published. Cheap, fast, and fully deterministic.

Open-Ended Questions: Let an LLM Be the Judge

When questions become open-ended (generate a piece of text, answer a question), things get complicated. One straightforward approach is string similarity matching with a threshold to determine if an answer is "close enough." But the more advanced approach is LLM as a judge — using another model to evaluate the output of the first.

JF emphasizes that having a judge LLM make subjective calls like "is this a good answer?" is unreliable. The truly effective technique is constraining the problem: break the evaluation into a set of clear true/false questions. Instead of asking "is this a good answer?", write a rubric: the answer I expect contains these 5 points — ask whether each one is true. This not only produces clear answers but also yields quantifiable metrics — like "3 out of 5 points hit, 60% accuracy."

In practice, these facts are sent as a JSON array to the judge model all at once, asking it to return a boolean for each — efficient with tokens and reliable. JF highlights a critical detail: these facts must never be provided to the Agent being tested. Otherwise, you're essentially handing it the answer key, and the Agent will take shortcuts and "cheat."

LLM as a Judge has become one of the dominant paradigms in AI evaluation in recent years, popularized by research from Stanford and other institutions. The core idea: since judging the quality of human language inherently depends on language comprehension, using a sufficiently powerful language model as an "evaluator" is theoretically more flexible than hard-coded rules. But this approach has a classic pitfall — position bias: judge models tend to score answers that appear earlier or are longer more favorably, regardless of content quality. The field has developed mitigation strategies like "swap order and average" and "multi-judge voting." JF's approach of "breaking into true/false questions" tackles subjective bias at the source: it converts the vague "good or not" into a list of verifiable objective facts, asking the judge model only to fact-check rather than make value judgments — dramatically reducing room for bias to creep in.

Practical Tips for Writing Reliable Evals

Drawing from his team's experience building Google Agent Skills, JF summarizes several validated principles:

• Follow RFC language conventions: Use clear language like must/must not/should in evaluation prompts to eliminate ambiguity and leave no room for interpretation in any judgment.
• Keep questions atomic: Don't pack two things into one question. "Is it JSON and does it contain a metadata field?" should be split into two separate true/false questions, each independently verifiable.
• Run multiple times and take statistical values: Because of non-determinism, the same Eval should be run 6 to 10 times, observing the distribution of correct rates (e.g., 50%, 60%, 60%, 50%), yielding more credible accuracy numbers along with standard deviation and error margin.
• Distinguish verbosity between "initial prompt" and "judge prompt": The initial prompt should reflect how a real user would actually ask — no need to over-specify (sometimes "will the Agent proactively choose this skill" is itself the evaluation goal). The judge prompt, on the other hand, should be as detailed and unambiguous as possible.

JF also revealed a counterintuitive finding: in 95% or more of cases, smaller models like Flash or Flash-Lite as judges produce consistent scores. In other words, the judging step often doesn't require expensive Pro-tier models — provided you polish your scoring rubric carefully and validate the judge with human annotation (manually label the expected true/false sequence first, then check whether the model produces the same result).

Who Needs Evals: Not Just Model Vendors

Many assume Evals are only for companies building models and Agent skills, but JF points out they're equally valuable for regular developers.

A typical scenario: measuring whether a skill is worth occupying context window space. Modern development environments may have large numbers of Skills and MCP servers installed — these consume tokens even when idle. Running a quick set of Evals — skill enabled vs. skill disabled — tells you: can the frontier model already handle this on its own? If the baseline accuracy already exceeds 90%, adding the skill on top may yield minimal benefit and could even mislead the model.

The live demo in the episode illustrated this perfectly: because the test environment didn't have gcloud installed, the version with the skill kept trying to run commands as the skill directed it — consuming far more tokens than the baseline. Meanwhile, the baseline model "knew it didn't have that tool" and simply switched to other resources to answer. This provides direct feedback for optimizing skill design.

Another counterintuitive finding: web search sometimes outperforms local skills. Because Evals test the "user goal/journey" rather than the skill itself, an Agent may find web search more convenient and bypass your skill entirely. The feedback you get: maybe you need to adjust the skill's name, description, or content so the Agent actually wants to call it. The team even tracks "whether the skill was actually used" as a standalone metric — by examining the Agent's trajectory (execution trace log) to see whether it activated and read the skill file.

Context Window refers to the maximum text length a large language model can process in a single inference, typically measured in tokens (roughly 1–2 tokens per English word, similarly for Chinese characters). Every piece of content loaded into context for each model call — system prompts, tool descriptions, conversation history — is billed and consumes inference time. MCP server or Agent skill description text, even if never actually invoked, occupies space the moment it's loaded into context. When a development environment integrates dozens of tools, these hidden costs accumulate quickly and can also interfere with the model's tool selection by adding too much "noise." This is the practical motivation behind JF's emphasis on using Evals to quantify whether a skill truly adds value, rather than relying on gut feeling.

Practical Considerations for Tool Selection and Maintenance Cost

For test harnesses, JF recommends several options: he personally uses Inspect AI most often; LangChain and Harbor also support this workflow; and for developers using Genkit or ADK (Agent Development Kit), Eval functionality is built directly into the framework. These harnesses handle the complex work of sandbox isolation (scalable with Podman, Kubernetes), multi-provider API integration, and more. Strip away the outer layer and the essence is: run Agent → give prompt → get response → evaluate response. He recommends trying a few and picking whichever "clicks" for you.

The episode also honestly addressed two real-world challenges. First, combinatorial explosion: the permutations of model × task × Agent can spiral out of control quickly, so you must stay focused — if you're upgrading a model, compare only the old and new models; if you're changing a prompt, hold all other variables constant. Second, maintenance cost: models are evolving rapidly (Gemini 3.8 Flash launched the day of the recording, just weeks after 3.7 Flash), and today's "new model" quickly becomes the "old model" — Eval prompts and facts need continuous updating. JF's advice: watch the trend, and once you notice accuracy rates rising across the board, make your tests harder to keep extracting useful signal.

For solo developers, JF also offers a balanced take: the return on investment depends on which Agents and models you plan to test and who your users are. If it's just for personal use, throwing hand-written skills into different Agents and observing reasoning differences across providers is genuinely interesting in itself — and a harness makes that kind of cross-Agent exploration much easier.

Inspect AI is an open-source LLM evaluation framework from the UK AI Safety Institute, designed to help researchers and developers systematically test model performance across a range of tasks — including safety, factual accuracy, and instruction following. It provides Task and Solver abstractions that support modular pipeline assembly, with multiple built-in Scorers. Genkit is Google's AI application development framework for JavaScript/TypeScript developers, with built-in flow tracing and Eval integration. ADK (Agent Development Kit) is Google's Python-ecosystem Agent building toolkit, which similarly treats Evals as a first-class citizen. Compared to building test scripts from scratch, the core value of these frameworks lies in handling the tedious engineering details of provider switching, concurrent execution, and result aggregation.

Closing: Make Every Token Count

JF's closing call to action is direct: go write Evals — stop testing by randomly running things in a terminal. Apply rigorous methods to test the AI tools and Agents you build. Follow the best practices the industry has developed: simple facts, atomic questions, objective and judgment-free, evaluate only what you asked for, and calibrate a trustworthy judge. Make every token you spend produce genuinely useful signal. That is the whole point of Eval-Driven Development.