Open-Source QA Agent in Practice: An End-to-End Testing Workflow That Tackles AI Hallucinations and Testing Bottlenecks

Two Core Contradictions in AI Coding

As AI-assisted programming penetrates enterprise development workflows, an uncomfortable question has surfaced: who actually verifies that the code AI writes is correct? A Bilibili creator known as "Ming Dui" recently open-sourced a QA Agent that his team has been refining internally — built specifically to address this pain point.

The first contradiction is AI self-evaluation hallucination. Ming Dui points out that the most common AI Coding scenario goes like this: the AI finishes writing code and tells you "it's done," you ask "are you sure?" and have it check again, and it still says "done." The problem is that you can't let AI be both the player and the referee. Self-assessment is inherently prone to hallucination, lacking any independent fact-checking mechanism.

The second contradiction is that testing can't keep pace with accelerated development. In traditional development workflows, the scheduling ratio between development and testing is roughly 2:1 — if frontend and backend development takes 10 days, testing takes 5. But AI Coding has boosted development efficiency by 3 to 10 times, compressing the development schedule to around 3 days, while the testing schedule can't shrink proportionally and still requires about 5 days. Testing has now become the new bottleneck.

AI self-evaluation hallucination is technically rooted in the autoregressive generation mechanism of large language models — when a model generates the conclusion "the code is complete," it relies on semantic coherence judgments about its own output rather than actual runtime verification. This resembles the psychological bias of subjective human confidence, but in LLMs it's more systematic: models tend to maintain contextual consistency and assign high confidence to content they've already generated, even when logical errors or edge-case omissions are present. The engineering approach to solving this is Separation of Concerns — the model that generates code and the mechanism that verifies code must be mutually independent, not sharing the same reasoning context, or the verification result will be contaminated by the "anchoring effect" of the generation process. QA Agent's design of introducing an independent Agent is fundamentally an application of this engineering principle to AI development workflows.

An End-to-End Pipeline Built from 8 Skills

QA Agent is designed to turn testing into a complete automated workflow. According to Ming Dui, it consists of 8 Skills covering 7 dedicated phases: context collection, risk analysis, test case design, script generation, execution and repair, code review, and report output. From reading the code to producing results, the entire pipeline runs fully automated in sequence — a human only needs to fill in the configuration once and confirm the test cases once.

Its core methodology is intervening as a third-party. QA Agent doesn't take the AI developer's word for it — instead, it works directly from the facts in requirement documents and code changes to identify business risks, define the testing scope, and ultimately let humans make the final call. Correctness is locked down by automated test scripts; delivery status is demonstrated through the factual evidence in automated test reports.

In Ming Dui's words: the conclusion isn't "I think it's fine" — every conclusion must map to an assertion. Whether coverage is actually achieved or not is what determines whether the process moves forward. This mechanism of converting subjective judgment into objective assertions is the key design that combats AI hallucinations.

The "end-to-end pipeline" here refers to a Directed Acyclic Graph (DAG)-style automated workflow: each Skill is an independent execution node, the output of upstream nodes serves as input for downstream nodes, and the entire chain requires no manual handoffs. This architecture in LLM applications is typically implemented using workflow orchestration frameworks (such as LangGraph, Dify, AutoGen, etc.), where each node can be an LLM call, a tool call, or a code execution sandbox. The "third-party intervention" is implemented by having the Agent's system prompt explicitly instruct it to act as an independent reviewer rather than a collaborator — with inputs sourced from code diffs and requirement documents rather than conversation history. This severs context sharing with the development-side AI at the data level, ensuring independent judgment.

Real-World Testing Uncovers Risks Human Test Cases Would Miss

The value of the workflow was validated in real-world testing. Ming Dui noted that QA Agent helped the team immediately discover risks in core processes — for example, whether experience points would be erroneously distributed when a program's transaction is rolled back. This kind of edge case is something testers would never think to write a test case for manually.

This shows that the value of an automated testing Agent isn't just speed — it's the systematic scanning of risk surfaces. Manual testing is constrained by experience and bandwidth, making it easy to miss hidden paths like transactions, concurrency, and abnormal rollbacks. An Agent conducting risk analysis based on code facts tends to achieve much broader coverage.

Accumulating a Test Case Library: The First Time Is a Cost, the Second Time Is a Return

Another noteworthy design aspect of QA Agent is continuous accumulation. It saves historical test cases over time, gradually building up a long-term test case library that becomes a core team asset.

Regression testing becomes extremely lightweight as a result: when the next version needs regression testing, just say "regress" in the Agent and it runs the existing scripts — no regeneration needed. It also supports adding new test scenarios: existing cases and scripts stay untouched, with only incremental additions of new cases and scripts.

Ming Dui puts it plainly: the first time you use it is a cost, the second time is a return. This also clarifies the tool's applicable scope — it's designed for enterprise-level projects that require long-term maintenance and continuous iteration. Across the entire workflow, humans only intervene at two points: filling in the configuration at the start and confirming the test cases in the middle. After that, it runs itself and you review the results. It takes repetitive labor off human hands while keeping final quality judgment with the responsible person.

The "test case library accumulation" here corresponds to the Regression Test Suite concept in software engineering. The traditional pain point with regression testing is that script maintenance costs grow linearly with each version iteration — manually written scripts frequently break after requirement changes, making the maintenance burden outweigh the benefits. QA Agent manages the script library through incremental additions rather than full rewrites: historically verified cases remain stable, and new scenarios are appended rather than overwritten. This is analogous to how version control systems manage code. From a cost structure perspective, the marginal cost of LLM-generated test scripts is far lower than manual writing, meaning the upfront investment of "building the library for the first time" can be recouped within a few iterations. This design gives test assets genuine compound returns — the more you use it, the lighter it gets, not heavier.

Applicable Scope and Open-Source Information

Ming Dui specifically cautions: for demo projects or one-off scripts, the overhead of this workflow will actually exceed the benefits. The reason is straightforward — there's an upfront cost to setting up the workflow and generating the test case library, and only through repeated regression and long-term iteration can those costs be amortized and the returns amplified.

This QA Agent is now open-source and can be installed with a single command. For teams pushing forward with AI-native development workflows while struggling with testing bottlenecks, it offers a viable example of how to automate and fact-ground the testing phase.

From a broader perspective, as AI Coding dramatically accelerates development, "who verifies AI's output" is becoming a new challenge for engineering efficiency. The approach taken by tools like QA Agent — using an independent Agent for fact-based verification, replacing subjective judgment with assertions, accumulating assets into a test case library — is very likely a necessary step on the path toward mature AI-native development workflows.