Three Stages of AI Large Model Testing: From Manual to API Automation in Practice

AI large models are reshaping how the testing industry works. A course on "AI large model testing" on Bilibili lays out a clear progression path: from basic content input, to prompt engineering, and then to code-based invocation via API. This path not only defines how testers collaborate with AI, but also reveals the two core directions for AI application deployment—RAG and Agent.

Stage One: Understanding the Essence of Large Models—"Text In, Text Out"

The fundamental principle of large language models can be simplified to a transformation process: a piece of text is input, processed through a pretrained model and self-attention mechanism, and another piece of text is output. This seemingly simple structure is precisely the technical foundation for translation and chat scenarios.

The course emphasizes that the smallest unit a model processes is the token (word segmentation), but in real projects there is no need to split manually, because different models come with their own tokenizers and rules vary. Understanding this helps testers realize: whether input is typed manually on a webpage or APP or entered through automation tools, it makes no difference to the model. The real dividing line lies in the "quality" of the input content—aimless chatting produces no business value, while clear instructions enable AI to analyze test points, design test cases, and even generate code.

Self-Attention is the core component of the Transformer architecture and the key to modern large language models' ability to understand long-text context. Its basic idea is: when processing each token in a sequence, the model calculates the association weights between that token and all other tokens in the sequence, thereby dynamically aggregating global information. For example, when the model reads the word "apple," the self-attention mechanism helps it determine whether the preceding text refers to "fruit" or "company," thereby resolving ambiguity. Unlike traditional recurrent neural networks that process words sequentially, self-attention can compute in parallel, greatly improving training efficiency and capturing long-distance dependencies. For testers, understanding this helps explain why models produce "hallucinations" or forget early information in long conversations—the allocation of attention weights is not perfect, and the model's context window also has length limits.

Stage Two: Prompt Engineering—Making AI Output Usable Test Assets

When input upgrades from "content" to "clear instructions," we enter the prompt engineering stage. The course gives an intuitive example: asking AI to "reply with no more than two characters each time"—the model still outputs content, but obeys the instruction constraint. This shows that the underlying mechanism of large models—"must generate content"—remains unchanged, but prompts can significantly influence the output form.

For testers, this means you can ask AI to play a specific role, output test cases in a required format, or limit the scope of analysis. However, this stage is still limited to the official webpage or APP—you cannot hide the logo of platforms like DeepSeek^Ad, nor is it easy to package capabilities into an independent tool for teams or customers.

Stage Three: API-Based Invocation—Opening the Door to Automation and Agents

The core of advanced usage is abandoning the official interface and invoking large model APIs through code. The course demonstrates using the OpenAI library as an example: instantiating the client requires an endpoint address and an API key. There is a noteworthy detail here—the difference between API key and token. Tokens usually have an expiration time, while API keys are designed without an expiration concept, making them suitable for long-term use in environment variables.

Code invocation uses the SDK approach rather than Postman or requests. When calling, you need to specify the model name, message role (user/assistant/system), and whether to enable streaming output. Streaming output returns content as it is generated, similar to the character-by-character display in web chat; disabling streaming waits for the complete result to be returned all at once. The course points out that this directly affects user experience testing—if you don't understand streaming transmission principles when testing AI applications, it's easy to misjudge a "stuck" interface as a failure.

More importantly, code invocation makes "AI generates code and automatically executes it" possible. Testers can have AI generate interface test code, save it as a file, and run it directly, forming a fully automated chain from test case design to execution. This is precisely the prototype of an Agent—AI designs test cases, AI executes, AI generates reports, and even automatically sends emails.

SDK (Software Development Kit) is a packaged code library provided by vendors for developers, typically including authentication, request construction, error handling, retry mechanisms, and other underlying details. The OpenAI library used in the course is a typical SDK approach: developers only need to instantiate the client, pass in the API key, and call methods like chat.completions.create, without manually assembling HTTP request headers, serializing JSON bodies, or parsing response status codes. Compared with sending HTTP requests directly via Postman or manually constructing with the requests library, the SDK's advantages are cleaner code, friendlier type hints, and better version compatibility guarantees. But note that SDKs update with API version iterations, and testers should pay attention to the matching relationship between the SDK version used and the model API version, avoiding situations where new parameters are not recognized due to an outdated SDK. In automated test scripts, it is recommended to encapsulate SDK calls as independent functions for unified management of timeouts, exceptions, and logs.

Insights from a Testing Perspective: Understand the Development Principles to Test AI Applications Well

The course repeatedly emphasizes one point: if you only do black-box testing at the outsourcing level and perform traditional functional testing on AI product webpages or APPs, there is no essential difference from ordinary Web testing. Truly valuable testing requires understanding the development principles of AI applications—how models are invoked, the difference between streaming and non-streaming, and the role of role parameters.

When testers can invoke large models through code, their perspective naturally turns to the two mainstream directions for AI application deployment: RAG (Retrieval-Augmented Generation) and Agent. The course argues that a simple chatbot does not count as an AI application, because models inherently possess conversational ability; what truly requires engineering packaging is knowledge base retrieval and automated task execution capabilities. This also explains why large models themselves tend toward being free, while solutions around RAG and Agent are where commercial value concentrates.

For practitioners hoping to enter the AI testing field, this three-stage path provides an actionable starting point: first understand the basic principles of text transformation, then use prompt engineering to improve output quality, and finally unlock automation and agent development capabilities through API invocation. A coding foundation is the threshold for the third stage, but the course also acknowledges that without a coding foundation, rapid mastery in a short time is impossible—knowledge of Python and API invocation needs to be acquired in advance.

RAG (Retrieval-Augmented Generation) is a technical architecture that combines external knowledge base retrieval with large model generation. Its basic process is: after a user asks a question, the system first retrieves document fragments most relevant to the question from a vector database, then splices these fragments as context into the prompt, and hands them to the large model to generate an answer. RAG solves pain points such as large models' fixed knowledge cutoff dates, inability to access private data, and tendency to fabricate facts, enabling enterprises to inject domain knowledge without retraining models. From a testing perspective, the quality of a RAG system depends on retrieval recall, the accuracy of context splicing, and the faithfulness of generated content to retrieval results—testing priorities differ significantly from traditional pure model testing.

Agent is a system that adds planning, tool invocation, and memory capabilities on top of large models. Unlike single-turn Q&A, an Agent can decompose complex tasks into multiple steps, autonomously decide which external tools to invoke (such as search, code execution, database queries), and adjust subsequent actions based on execution results. When testing an Agent, in addition to verifying the correctness of single-step outputs, you also need to focus on whether the task chain is complete, whether tool invocation parameters are valid, whether exception branches can correctly fall back, and state consistency across multi-turn interactions. This is the deeper reason why the course emphasizes that "understanding development principles is necessary to test AI applications well."

One-line Summary

A Bilibili course presents a three-stage progression path for AI large model testing, revealing the two major deployment directions of RAG and Agent.

Paragraph Summary

The article outlines the core content of an AI large model testing course, presenting a three-stage progression path for testers collaborating with AI: Stage one is understanding the essence of large models—"text in, text out"—and recognizing that input quality determines business value; stage two enters prompt engineering, using clear instructions to make AI output usable test assets, though still limited by official interfaces; stage three is code-based invocation via API, unlocking streaming output testing, automated execution, and agent development capabilities. The article emphasizes that if testers only do black-box functional testing, it is no different from ordinary Web testing; truly valuable testing requires understanding development principles such as model invocation, streaming versus non-streaming differences, and role parameters. Ultimately, the perspective should turn to the two major AI application deployment directions—RAG knowledge bases and Agent intelligent agents—which is also where commercial value concentrates.