Qt App Automation Testing: Injecting Business Knowledge into Agents with Skills

In agent-driven automation testing, a common pain point is that while an agent can operate a graphical interface, it knows nothing about the business rules of the application under test. For example, if you ask it to test a book search feature, it doesn't know which bookstore should be selected by default or how many results to take—it can only "guess" based on UI elements. The solution shared by a Bilibili creator uses a Skill mechanism to codify business knowledge so that the agent automatically loads and follows rules during testing. This is worth referencing for Qt application testing teams.

Why Agents Need Business Knowledge

The mainstream approach for agent-based UI automation testing is the exploration mode—identifying controls based on interface structure and simulating clicks. According to the video demo, when the agent operates an app called Calibrate, it can find and click the "Online Book Search" entry, but it doesn't understand the implicit rules behind this business scenario: what types of search keywords exist, how to choose a bookstore, or how many results to take.

This state of "knowing how to operate but not understanding the business" leads to a deviation between test cases and real user behavior. The video author points out that real users often don't have a very clear behavioral path when searching. If the agent can combine business knowledge to perform composite operations, testing will be closer to actual usage scenarios.

Exploration mode is the current mainstream paradigm for agent-driven UI automation. Its core is to let the agent infer executable actions by perceiving the interface structure (such as the accessibility tree of controls, coordinates, text labels) without relying on pre-written fixed scripts. Such methods typically combine vision-language models or interface parsers so that the agent can "look and click" on unfamiliar interfaces. However, exploration mode inherently lacks domain priors: it can only recognize "this is a button" but cannot judge "whether this button should be clicked in the current business scenario." Therefore, when tasks imply business rules (such as default options, data ranges, operation sequences), pure exploratory agents tend to produce seemingly reasonable operations that don't conform to business conventions. This is the fundamental reason why external knowledge injection is needed.

Skill Mechanism: Writing Business Rules as Agent-Readable Documents

The core of the solution is a .qk/skill folder located in the user directory. According to the structure demonstrated in the video, a separate subdirectory can be created for each application under test, for example, an online_search skill for the Calibrate app.

In this skill directory, key elements of the business scenario are described in document form:

• Search keyword types: book title, author, keyword
• Bookstore selection rules: follow explicit specifications when given; use the default bookstore when unspecified
• Result count convention: return the agreed number of results (10 in the demo) when not explicitly required
• Operation flow: the sequence of steps from entering the page to obtaining results

According to the author, after defining this, the agent will first read the skill document before executing the task, understand the business context, and then operate.

The Skill mechanism draws on the "tool/skill" abstraction in agent frameworks: it encapsulates a set of domain knowledge, operational constraints, and process descriptions as independent, reusable document units that the agent loads on demand before task execution. Unlike hardcoding logic in scripts, skill documents typically use natural language or semi-structured formats, making them easy for humans to maintain and for agents to parse. The key design lies in "on-demand retrieval"—the agent matches relevant skills based on the current task semantics rather than loading all knowledge at once, thereby avoiding context overload. This pattern is especially valuable in automation testing because it decouples "the business rules of the application under test" from test code, forming an independently evolvable knowledge base. When the interface or process changes, only the corresponding skill document needs to be updated.

Hands-On: The Complete Execution Chain Behind a Vague Instruction

The video validates with a vague instruction: "Help me find some e-books suitable for learning Python in Calibrate, just show me the results." This instruction specifies neither the keyword type nor the bookstore.

The agent's execution process demonstrates the value of skills:

1. Load skill: automatically read the online_search skill document and understand the rules
2. Check environment: discover that Calibrate is not running and automatically launch the app
3. Locate entry: find the "Online Book Search" control and open it
4. Apply default rules: since the user didn't specify a bookstore, deselect other bookstores and keep only the default one
5. Execute search: identify keywords and submit the search
6. Verify results: obtain 16 results, review them one by one, and close the interface

Ultimately, the agent completes a composite operation that conforms to business specifications rather than mechanically clicking interface elements.

Two Insights for Testing Practice

Knowledge codification is more flexible than script reuse. Traditional automation testing hardcodes business logic in scripts, and everything breaks when the interface changes. The Skill approach separates business rules from code into natural language documents that the agent reads in real time at each execution. Adjusting rules only requires changing the document, not the script.

Vague instruction testing is closer to real users. The author emphasizes that user behavior is often imprecise. Letting an agent handle vague instructions with business knowledge can cover scenarios like "how the system falls back when the user isn't clear," which step-by-step test cases struggle to simulate.

Vague instruction testing refers to inputting incomplete, imprecise, or ambiguously interpretable operation requests to the system under test and observing whether the system can give a reasonable response based on default rules or fallback logic. Such testing is difficult to cover with traditional scripted cases because scripts usually require fully determined steps and parameters. An agent with business knowledge can autonomously fill in context under vague input (such as selecting the default bookstore or agreeing on the number of results), thereby simulating system behavior when real users "aren't clear." This testing approach helps discover boundary issues such as missing default values, improper exception handling, and conflicting business rules. It is a useful supplement to precise test cases, but note that its reproducibility depends on the stability of the agent's decision-making.

Applicability Boundaries and Open Questions

It should be noted that the above content is based on a demo from a single video source. Details such as the specific format specification of skills, the upper limit of the agent's parsing capability, and the expressiveness of complex business rules were not elaborated in the video. For applications with complex business rules and many state dependencies, whether pure document-based skill descriptions are sufficient still needs validation in actual projects. However, the idea of "supplementing agents with domain knowledge through readable documents" provides a low-cost, easy-to-maintain path in Qt applications and even broader UI automation testing.

One-line Summary

A Bilibili creator shares how to inject business knowledge into agents using Skill documents to improve the realism of UI automation testing.

Paragraph Summary

The article introduces a Skill mechanism for agent-based UI automation testing: addressing the pain point that agents "know how to operate but not the business," it creates skill documents for each application under test in the .qk/skill folder in the user directory, describing search keyword types, bookstore selection rules, result count conventions, and operation flows in natural language. The agent automatically loads the corresponding skill before executing tasks, thereby completing composite operations such as launching the app, applying default rules, searching, and verifying results under vague instructions. The author believes this approach separates business logic from scripts, requiring only document changes for rule adjustments, lowering maintenance costs, and that vague instruction testing is closer to real user behavior. However, the solution is based on a single video demo, and its applicability to complex business scenarios still needs project validation.