From Principles to Practice: A Beginner's Guide to AI Large Models and LLM Evaluation

For QA engineers and developers, AI large models are no longer a distant technical buzzword. If you can type and chat, you can use AI — this remarkably low barrier to entry is reshaping how software testing gets done. This article, based on content shared by Mashibing Education on Bilibili, outlines the technical landscape of AI large models, their core principles, and how testers can use them effectively.

What Exactly Is an AI Large Model

Many people treat "AI large model" as a single unified concept, but strictly speaking, that's not quite right. AI is the umbrella term for artificial intelligence, and large models are just one way of implementing it.

AI development goes back quite a long way — relevant papers appeared as early as the 1970s, primarily grounded in statistics and rule-based strategies. What we talk about today — DeepSeek^Ad, Gemini, ChatGPT — falls under the category of modern artificial intelligence.

In terms of the technical hierarchy, the lineage looks roughly like this: Modern AI → Machine Learning → Deep Learning → Large Language Models. Machine learning encompasses reinforcement learning and deep learning, and AI large models are a branch of deep learning. In the narrow sense, when people say "AI large model," they're really referring to a Large Language Model (LLM).

It's worth mapping out the three mature directions deep learning has excelled in over the past eight years: computer vision (OpenCV), speech recognition, and natural language processing (NLP). The face-slimming and eye-enlarging filters on Douyin are computer vision. WeChat's voice-to-text and smart speakers are speech recognition. And products like ChatGPT and DeepSeek — where you type and get a response — are fundamentally natural language processing.

Why Large Language Models Took Off

Computer vision is actually a very mature field — image recognition has become remarkably good. But both image and speech AI share a common weakness: they're not easy for the general public to use. No matter how good speech recognition gets, it won't help you if the AI speaks a language you don't understand. Computer vision can interpret images, but it still has to communicate results through text.

It wasn't until language models (text models) became good enough that the landscape truly changed. People could tell the AI what they needed in plain text, and the AI could respond in plain text. A straightforward conclusion from the shared content: if you can chat with text, you can use AI.

This also explains why AI seemed to "suddenly" explode in popularity around the Lunar New Year — it didn't appear overnight, but rather crossed a threshold after years of accumulation. Earlier AI capabilities were more domain-specific, accessible only to specialists in those fields. Natural language processing changed that, allowing anyone to use AI through ordinary conversation and dramatically lowering the barrier to entry.

That said, a low barrier doesn't mean everyone uses it well. Why can some people get high-quality output from AI while others go back and forth without getting anywhere? Prompt engineering techniques are real, and understanding the underlying principles is what makes those techniques click.

The Core Principles of Transformer

To truly understand large models, you can't avoid the Transformer. It's often translated as "变形金刚" (Transformers, as in the movie robots), but "converter" or "transformer" is actually a more accurate description of what it does.

The shared content uses a simplified translation example to illustrate how it works. Take translating "I love you" into Chinese ("我爱你"):

• Encoding: The English characters are encoded into vectors — they're no longer letters but mathematical vectors;
• Vector space computation: Once in the vector space, the system looks for other vectors that are semantically similar. For instance, "我爱你" (I love you), "我喜欢你" (I like you), "我稀罕你" (I adore you), and "我中意你" (I fancy you) are semantically close, so they cluster near each other in the vector space;
• Decoding and probability inference: The model repeatedly calculates the highest-probability output. It first settles on "我" (I), then infers "爱" (love) over "喜" (like) because the probability is higher, and finally "你" (you) — generating the result one token at a time.

This process reveals the fundamental nature of large models: they are probabilistic. That means if you ask the same question a hundred times, you might get a hundred slightly different answers. It's exactly why AI responses vary each time you ask the same thing.

The Transformer architecture was introduced by Google in 2017 in the paper Attention Is All You Need, and it fundamentally transformed natural language processing. Its core innovation is the Self-Attention mechanism: when processing a piece of text, the model calculates the relevance weight between every word and every other word, capturing long-range semantic dependencies. For example, in the sentence "She put the apple in the basket because it was too heavy," the model can determine whether "it" refers to the apple or the basket.

The previous dominant approach, RNNs (Recurrent Neural Networks), could only process text word by word in sequence — slow and prone to forgetting distant context. Transformer processes the entire sequence in parallel, dramatically improving training efficiency and making it feasible to train massive models on enormous datasets. ChatGPT, DeepSeek, and similar products are all built on this architecture. Understanding this helps explain why models sometimes seem to "forget" earlier parts of a long conversation — it's directly related to the model's context window size limit.

How the Probabilistic Nature Defines Strengths and Weaknesses

Once you understand that large models are fundamentally probabilistic, you can reason through where they should and shouldn't be used.

Where They Excel

The probabilistic nature brings randomness and variety — which happens to be exactly what creative writing calls for. It's like talking to a person: if they always responded with the same few words, it would be boring. The probabilistic output, different every time, actually makes for more engaging interaction. Scenarios requiring creativity and diverse expression are where large models shine.

Where They Struggle

Precise computation is a clear weakness. Because the model isn't doing strict mathematical calculation, it can handle 1 + 1 just fine, but give it a system of equations and it may not outperform a calculator. The early trend of testing ChatGPT or DeepSeek with complex math to gauge their "intelligence" was essentially testing them on their weakest ground.

Authoritative, accurate answers also warrant caution. Take "what medication should I take for diarrhea" — a large model can only offer a reference based on symptom-medication associations it has seen, with no guarantee of accuracy or authority. That's why in domains like law, finance, and medicine — where errors are unacceptable — models are trained to avoid overly confident answers and instead guide users toward professional doctors, lawyers, and other qualified experts.

The limitation around "authoritative accuracy" also ties directly into a common phenomenon: hallucination. During the probability inference process, models sometimes generate content that sounds plausible but is factually wrong — fabricated paper citations, invented historical details, incorrect API function signatures. This isn't a bug; it's an unavoidable byproduct of the probabilistic generation mechanism. The model is fundamentally predicting "what word most likely comes next," not retrieving facts from a database.

For QA engineers, this means: test cases, test data, or interface descriptions generated by AI must go through human review before being used. They cannot be taken at face value. This is especially true when AI output involves specific version numbers, function names, or field names — cross-verification is essential. This is the most direct engineering takeaway from understanding hallucination.

Applying AI in Testing — and Where to Watch Out

When you map large models back to testing work, the alignment between strengths and use cases becomes very clear.

Testing tasks where AI fits well: requirements evaluation and test case design. Testing inherently requires thinking about all kinds of unusual, unexpected edge cases — and AI's probabilistic, divergent nature is well-suited for this, surfacing scenarios humans might overlook.

Tasks where AI doesn't fit well: debugging errors. When asked to debug a bug, AI tends to latch onto a wrong hypothesis and dig in deeper rather than stepping back. Analyzing results is more manageable (since there's something to reason about), but pure fault localization isn't its strong suit.

For testing professionals, AI large models offer value on two levels: first, direct personal use — generating test cases, reviewing requirements, assisting with data analysis; second, as more and more tech companies and startups build AI-powered products (AI IDEs, intelligent customer service, AI language tutors, etc.), QA engineers need the ability to test these AI-based applications. Understanding the principles and knowing the boundaries is the first step into this domain.