MCP Protocol-Driven AI Testing Agent: Automated Execution, Diagnosis, and Reporting

When Tests Fail, AI Can Tell You Why

Traditional automated testing frameworks are good at "finding problems" but not at "explaining problems." When developers get a red failing test case, they often have to manually dig through logs, compare historical results, guess the root cause, then fix, rerun, and confirm. This cycle consumes a lot of time that could otherwise be spent writing business code.

A demo from an academic project proposes a combination approach: use the Model Context Protocol (MCP) to expose testing execution tools to AI clients, then let a local large language model step in to diagnose test failures, and finally generate timestamped structured reports and compare them with historical results. The system is named "MCP-Empowered AI Agent for Automated Test Execution, Failure Diagnosis, and Intelligent Reporting."

Its core proposition is clear: a testing system should not stop at "detecting errors"—it should help "explain failures" and "track the repair process."

System Architecture: MCP as the Middle Layer

According to the project's presentation, the entire system uses Python 3.10 as the primary language for application logic and integration. The key design is introducing MCP as a standardized communication layer between the AI client and the testing server, and exposing testing execution tools through FastMCP. Actual test case execution is handled by PyTest, with results stored in a structured JUnit XML format.

The AI diagnosis component does not rely on cloud APIs; instead, it uses Ollama as the local LLM runtime, with Llama 3.2 3B as the model. Report output supports both JSON and Markdown formats, and the project itself uses Git and GitHub for version control.

This technology choice sends a signal: the entire solution pursues localization, reproducibility, and low dependencies. Running diagnosis with a small 3B-class model means it can run on an ordinary development machine without calling paid APIs for every test failure. The trade-off is that diagnostic depth may be limited, but for the goal of "pointing out possible causes," whether it is sufficient depends on project complexity.

MCP (Model Context Protocol) is an open protocol proposed by Anthropic in 2024, aimed at providing a standardized communication interface between AI applications and external tools and data sources. It can be compared to a "USB-C port" for the AI world: whether the underlying system is a database, file system, or test executor, as long as it implements the MCP server specification, AI clients can discover and invoke these capabilities in a unified way. MCP uses a client-server architecture, transmits messages via JSON-RPC, and supports three types of primitives: tools, resources, and prompts.

FastMCP is a Python implementation library for the MCP protocol. It simplifies server definition and allows developers to quickly expose ordinary Python functions as MCP tools in a decorator style. In this scenario, FastMCP is responsible for wrapping PyTest's test execution capability into a tool callable by AI clients, thereby eliminating the need to write separate adapter layers for each testing tool.

Ollama is an open-source local large language model runtime that supports one-click pulling and running of various open-source models (such as Llama, Mistral, Gemma, etc.) on macOS, Linux, and Windows, and provides a local API compatible with the OpenAI style. Its core value is enabling model inference entirely locally, with no cloud call costs and no need to send sensitive data such as test logs externally.

Llama 3.2 3B is a small-parameter model released by Meta in 2024, with about 3 billion parameters. After instruction tuning, it is suitable for dialogue and text analysis tasks. A 3B-class model has low VRAM and memory requirements and can run smoothly on an ordinary development machine, but on complex tasks requiring multi-step reasoning or domain knowledge, its analysis depth is usually inferior to larger-parameter models. This explains the judgment in the article that "diagnostic depth may be limited."

Workflow: A Closed Loop from Submission to Report

The system workflow can be divided into several stages. After a user submits a test request, it is sent to the server via the MCP client. The server calls PyTest to execute test cases, and the test runner returns results and parses the XML.

There is a branch logic worth noting here: if all test cases pass, the AI diagnosis step is skipped entirely; only when failures occur does the local language model analyze the failure information and provide possible causes. This "on-demand triggering" design avoids unnecessary model calls and makes responses faster in passing scenarios.

Finally, the system generates a timestamped report, compares it with historical reports, and produces a human-readable summary. The value of historical comparison lies in answering a question that automated frameworks usually do not answer: is this failure new, or did it exist before and has now finally been fixed?

JUnit XML is a widely adopted test result exchange format. It originated from Java's JUnit framework and later became a de facto cross-language standard. PyTest can output test results in this format through plugins, including the name of each test case, its class/file, execution time, pass/fail/skip status, and exception type and stack trace information when failing.

The advantage of choosing JUnit XML as the intermediate storage format is decoupling: the test executor is responsible for producing structured results, and the AI diagnosis step only needs to parse this standard format without coupling to PyTest's internal objects. This also leaves room for future integration with other testing frameworks (such as unittest, NUnit)—as long as they can output JUnit XML, the diagnosis pipeline can be reused.

Experimental Validation: Controlled Failures and Historical Tracking

The project conducted two sets of experiments using a sample Python application containing four automated test cases.

The first set was a successful run: all four test cases passed, with zero failures, and the system correctly identified and skipped AI diagnosis. The second set was a controlled failure experiment: the team temporarily modified the discount calculation logic, artificially introducing an error, causing discount-related test cases to fail because the calculated value did not match the expected value. After analysis, the local language model located the problem in the percentage discount formula. After restoring the correct implementation and rerunning, all four test cases passed again.

The third set was historical analysis: the system compared 15 historical test reports, verifying the transition of the discount test case from failure to pass, thereby confirming that the failure had been resolved.

This set of experiments is small in scale, but its design intent is clear—it aims to prove not how high the diagnostic accuracy is, but that the entire "execution—diagnosis—tracking" pipeline can operate as a closed loop. The controlled failure experiment is particularly important because it provides a control with a known answer, allowing the model output to be tested.

Limitations and Next Steps

Based on the demo content, this system still has obvious room for expansion. Future directions mentioned by the team include: integrating with GitHub Actions so that tests are automatically triggered when code changes; supporting more testing frameworks rather than only PyTest; and further improving the depth of failure analysis.

It should be viewed objectively that the single-source demo did not disclose quantitative metrics for diagnostic accuracy, and there is a lack of data supporting the 3B model's performance in complex failure scenarios. Historical comparison is currently based on 15 reports, which is still a small scale. These are all steps that must be taken to move from "demo-usable" to "engineering-reliable."

However, the combination of MCP + local small models + testing toolchain itself has reference value: it places AI in the "explanation layer" rather than the "execution layer" of the testing process, letting deterministic tools do deterministic things and letting the language model handle fuzzy attribution problems. For teams exploring AI-assisted R&D workflows, this is a cost-controllable starting point.

One-line Summary

Use MCP to connect testing tools with a local small model, automatically diagnosing test failures and tracking fixes.

Paragraph Summary

This project proposes a combination of "MCP + local small model + testing toolchain," aiming to extend automated testing from "finding problems" to "explaining problems." The system uses Python as its main language, exposes PyTest execution tools to AI clients via the MCP protocol through FastMCP, and stores test results in structured JUnit XML. AI diagnosis uses the Ollama local runtime with the Llama 3.2 3B model, triggered only when failures occur and skipped entirely in passing scenarios. The system ultimately generates timestamped JSON/Markdown reports and compares them with historical reports to track whether failures have been fixed. The experiments include three sets: successful runs, controlled failures, and historical analysis, validating the closed-loop operation of the "execution—diagnosis—tracking" pipeline. However, quantitative data on diagnostic accuracy and performance in complex scenarios is still lacking.