How to Test WebMCP Tools? Using Evals to Assess Non-Deterministic AI Behavior

Why Traditional Testing Falls Short

WebMCP tools are essentially JavaScript code, so you can absolutely write unit tests and integration tests—that's fine. But these tests only verify one thing: whether the code runs reliably under deterministic inputs.

The real challenge lies with the caller. Your tools will be invoked by LLM-based AI Agents, and Agents are non-deterministic. For the same user intent, the Agent may sometimes select the correct tool and sometimes not; sometimes it passes the right parameters, sometimes the wrong ones; for user journeys that require consecutive calls to multiple tools, the Agent may go off track midway.

This raises a key question: Will the Agent select the right tool? Will it pass the right parameters? Can it complete a user journey that requires multi-step tool calls? These are dimensions that unit tests cannot cover, and they require a different approach—Evals.

Evals: Writing Tests for Non-Deterministic Systems

Evals is a testing methodology specifically designed for non-deterministic systems. Its core idea is not to assert "the output must equal some value," but to define "the expected tool call sequence" and then let the Agent actually run through it to see whether it meets expectations.

Specifically: for a particular user journey, you define the expected tool calls in a JSON file. This JSON is your evaluation case—it describes not the code logic, but the behavior the Agent should exhibit.

Once you have the case, you can choose between two modes to run the evaluation.

Non-deterministic systems refer to systems that may produce different outputs given the same input. Traditional software is deterministic: given the same input, the output is constant, so precise assertions can be used for testing. But LLM output is inherently probabilistic sampling—even with identical prompts, the generated token sequence may differ each time. This means that when testing LLM Agents, "a single run failed" does not mean the system is broken, and "a single run succeeded" does not mean the system is reliable; you can only assess it by statistically measuring the success rate across multiple runs. Evals is designed precisely for this characteristic: it does not pursue exact matching in a single run, but focuses on whether the distribution of the Agent's behavior across multiple samples matches expectations.

Expected tool call sequence (expected tool calls) is the core structure of an evaluation case. It breaks a user journey into an ordered chain of tool calls, recording the tool name and key parameters at each step. During evaluation, the system compares the Agent's actual call trajectory against this sequence and computes the degree of match. Unlike asserting the final output, this comparison focuses on the process—whether the Agent called the correct tools in a reasonable order, and whether the parameters fall within the expected range. Therefore, even if the Agent ultimately gives the correct answer, if it took detours or called the wrong tools along the way, the evaluation will still flag it as not meeting expectations, thereby exposing problems in tool design or descriptions.

Two Evaluation Modes: browser and local

Browser Evals: Run Directly Against the Real Website

The most direct approach is to run the evaluation against the real WebMCP tools on your website, known as browser evals mode. It is close to the production environment and can verify how the tools perform in the real page context.

However, the official recommendation is: before running browser mode, first conduct tests in an isolated environment.

Local Evals: Isolate First, Then Integrate

Local evals mode does not depend on the website; it tests only the tools themselves. You need a schema JSON file to mock your website's WebMCP tools and their input schemas.

This schema file is not just auxiliary material for testing—it is actually a very good starting point. You can think of it as evals-driven development: first define the tool interface and expected behavior clearly, then write the implementation.

Mock tools and schema JSON: In local mode, the schema JSON file plays the role of the tool contract. It declaratively describes each WebMCP tool's name, functional description, and the JSON Schema of its input parameters (types, required fields, value ranges, etc.). Based on this, the evaluation framework constructs a virtual environment that lets the Agent "see" these tools and make choices without access to the real website. The benefit is isolation: interface-level issues such as how tools are exposed, the wording of descriptions, and parameter definitions are tested separately, without interference from environmental factors such as page loading or network conditions. If the Agent cannot correctly select tools in local mode, the problem lies in the tool definitions themselves; conversely, if local passes but browser fails, the problem is more likely in the real page context or the way tools are exposed.

Evals-driven development borrows the idea of test-driven development (TDD), but the object shifts from deterministic code to Agent behavior. Traditional TDD writes a failing unit test first and then the implementation; evals-driven development instead writes the schema JSON and expected call sequences first, clarifying "what the tool interface looks like" and "how the Agent should use it," and then implements the WebMCP tools. The benefit is that interface design is forced to think from the Agent's perspective from the very beginning—whether the tool name is clear enough, whether the description is sufficient for the LLM to understand its purpose, whether the parameters are easy to fill in correctly—rather than fixing things after the implementation is complete.

Running an Evaluation: Only Multiple Runs Are Convincing

The Chrome team provides an open-source experimental tool, WebMCP eval-cli, which you can use directly to run evaluations. The workflow is roughly: select a model, set the number of runs (set to 5 in the example).

The number-of-runs setting is critical. Because LLM output itself fluctuates, a single pass does not prove anything; you need multiple runs to build confidence in the results. The more runs you do, the more trustworthy the pass rate metric becomes.

After the evaluation suite finishes, you get a detailed report including:

• Pass rate: the proportion of successful runs by the Agent across multiple runs
• Agent trajectories: what the Agent actually did at each step, making it easy to pinpoint where it deviated from expectations
• List of available tools: which tools the Agent could see at each step

The last item is particularly useful for debugging—if a tool is not visible at a certain step at all, the Agent naturally will not call it, and the problem is not in the Agent's selection logic but in how the tool is exposed.

Pass rate is the most core quantitative metric in Evals, defined as the proportion of multiple runs in which the Agent successfully completes the expected call sequence. Because LLM sampling is random, a tool with a single-run success rate of 50% may fluctuate greatly across multiple runs, so the pass rate must be interpreted in conjunction with the number of runs: passing 3 out of 5 runs and passing 30 out of 50 runs both yield 60%, but the latter provides much higher confidence than the former. In practice, the more runs, the more stable the estimate of the pass rate, but the higher the cost. It is recommended to first use a small number of runs to rapidly iterate on the schema and tool descriptions, and then increase the number of runs near release to obtain reliable conclusions.

Agent trajectories record the Agent's complete decision process in each run: which tools it saw at each step, which tool it selected, what parameters it passed, and what results it obtained. The value of trajectories lies in pinpointing the cause of failure—the pass rate only tells you "how many times it failed," while trajectories tell you "why it failed." Common failure modes include: selecting the wrong tool (usually due to insufficient distinction between tool descriptions), filling in parameters incorrectly (unclear schema constraints), deviating midway through a multi-step journey (lack of intermediate state hints), and a tool being entirely invisible at some step (a problem with how the tool is exposed). Combined with the list of available tools at each step in the report, you can quickly distinguish between "Agent selection logic problems" and "tool exposure problems."

Try It Yourself

The value of this methodology is that it turns "whether the Agent can use your tools well" from a subjective feeling into a quantifiable metric. Traditional testing answers "is the code correct," while Evals answers "does it work smoothly for the Agent"—the latter is the scenario that WebMCP tools truly target.

The tool is already available on GitHub and NPM, and the repository includes examples you can run directly. It is recommended to start by defining a schema JSON, write a few local-mode evaluation cases first, and then gradually transition to browser mode.

One-line Summary

WebMCP tools need Evals to test the non-deterministic invocation behavior of LLM Agents.

Paragraph Summary

The article points out that traditional unit tests and integration tests can only verify the stability of WebMCP tool code under deterministic inputs, but cannot cover the non-deterministic behavior of LLM Agents when invoking tools—the Agent may select the wrong tool, pass the wrong parameters, or deviate during a multi-step journey. To address this, the article introduces Evals: using JSON files to define the expected tool call sequence for a specific user journey, letting the Agent actually run and comparing the results. Evaluation is divided into local mode (using schema JSON to mock tools and test the tools themselves in isolation) and browser mode (running against the real website), with the recommendation to do local first and then browser. The article also introduces the open-source tool WebMCP eval-cli, emphasizing the need for multiple runs to build confidence; the report includes pass rate, Agent trajectories, and the list of available tools at each step, and advocates an evals-driven development approach starting from schema JSON.