Understanding LLM Agents from Scratch: Why Agent Tuning Matters

Why Large Language Models Need Agent Technology

Ask a large language model "How old is Andy Lau right now," and it might dig up a stale webpage and return an age from some past year — a classic limitation of pure LLMs. This Bilibili tutorial series on LLM application development cuts straight to real-world scenarios to explain why Agent technology exists and why it matters.

The first category of use cases the course highlights is telling: looking up the origin of a classical Chinese phrase, asking follow-up questions about time and weather, querying real-time sales figures, or asking about traffic conditions while driving. What these questions share is a need for real-time information or external knowledge sources — something a base LLM simply cannot provide.

An LLM's training data is a static snapshot collected up to a certain point in time. Updating it is extremely costly, and the model has no live connection to the real world. The course distills the core reasons into three points: first, hallucination is nearly impossible to eliminate at the model level, since LLMs are fundamentally probabilistic generators and deviation is unavoidable; second, model parameters cannot be updated in real time — the model is static; third, complex business workflows require task decomposition, multi-step execution, and interaction.

This is where Agents add value: they don't just let an LLM "talk" — they let it act. By calling external tools like search engines, calendars, and weather APIs to fetch the latest information and then feeding that into the model's reasoning, Agents unlock value across a far broader range of business scenarios.

The Key Difference Between Agents and Traditional Applications

Compared to the standard ChatGPT-style single-turn Q&A, the most notable distinction of an Agent is process transparency. It surfaces intermediate reasoning steps and tool-call results, making answers traceable and verifiable.

This directly addresses the trust problem with LLMs as "black boxes." As the course points out, users often hesitate to fully trust an LLM's answers and feel compelled to verify them independently. By laying out the reasoning chain, Agents both improve reliability and make it easier for users to audit the output.

Furthermore, requests like "Book me a flight to Shanghai this Friday" or "Schedule a requirements review meeting with the product team from Wednesday to Friday this week" typically can't be completed in a single step. They require multi-step execution with intermediate interactions — after pulling up multiple flight options, the system needs confirmation on which one to pick, and then proceeds to payment and booking. This multi-step, interactive task pattern is simply beyond what the traditional single-turn Q&A model can handle.

RAG (Retrieval-Augmented Generation) is an important intermediate step in understanding how Agent technology evolved. The core idea is to retrieve relevant text snippets from an external knowledge base or index before the LLM answers, and pass those snippets as context alongside the query — giving the model something concrete to ground its response in. Unlike Agents that call external tools directly, RAG typically retrieves from a static vector database. Organizations pre-vectorize internal documents, product manuals, FAQs, and similar materials; at query time, RAG uses semantic similarity to surface the most relevant passages. This mechanism effectively reduces hallucination, but it has clear limitations for scenarios requiring real-time data (e.g., current stock prices, today's weather) or multi-step operations (e.g., placing an order, sending an email) — which is precisely why Agents need to go further with an expanded toolset and planning capabilities.

The Technical Architecture of an Agent

The course references a summary diagram from an overseas site to map out the evolution, and the progression is clear:

• Base LLM: Prompt in, answer out — a pure black-box Q&A system.
• RAG (Retrieval-Augmented Generation): For questions requiring real-time or precise information, first retrieve relevant context from an index, knowledge base, or the open web, then pass it along with the question to the LLM — reducing hallucination.
• Agent: A significant expansion on top of RAG, featuring a richer toolset, short- and long-term Memory, and reasoning/planning capabilities.

The most critical component in this architecture is Planning: it encompasses deep thinking, self-critique, and chain-of-thought reasoning, progressively decomposing complex problems into subtasks and executing them iteratively until the task is complete. On the left side sits a rich toolset; paired with short- and long-term Memory, the system issues Action commands to invoke those tools. Together, this combination makes Agent a complete technical framework ready for production deployment.

Why We Still Need Agent Tuning

If you can implement an Agent just by writing complex Prompt Engineering instructions (like AutoGPT does), why bother with dedicated training? The course treats this as its second major topic.

The key prerequisite is — the model needs to be smart enough. Writing a Prompt is essentially handing the LLM a "user manual" that spells out the workflow and requirements. But in practice, only GPT-4-class models and above can reliably follow complex Prompts to complete sophisticated tasks — models including GPT-3.5 and below generally can't. Feed a smaller, less capable model the kind of elaborate Prompt that AutoGPT uses, and it typically fails to understand it.

Real-world constraints compound the problem: many business environments can't call the most powerful model APIs — they require private, on-premises deployment (intranet), which rules out closed-source models like GPT-4, GPT-4o, and o1. At the same time, teams need to keep inference costs under control. These constraints often force teams to build and host their own models.

In practice, Agent Tuning — specialized fine-tuning for Agent tasks — has proven capable of giving small models like 3B and 7B reasonably strong Agent capabilities, with lower inference costs and easier deployment. The course uses a fitting analogy: it's like a college graduate entering the workforce — a general education provides broad capabilities, but on-the-job training with real cases is still needed before they can hit the ground running.

AutoGPT was an open-source Agent framework that drew widespread attention in early 2023. Its core idea was to use a carefully designed system Prompt to get GPT-4 to autonomously decompose goals, call tools, reflect on execution results, and loop iteratively until a task was complete. It was the first product-form demonstration to the public that "letting an LLM autonomously complete complex tasks" was possible, spawning a wave of subsequent Agent frameworks and experiments. However, AutoGPT's real-world use also exposed the limits of pure Prompt-driven approaches: once tasks become sufficiently complex, models easily fall into loops, generate invalid actions, and are highly sensitive to base model capability — GPT-4 could barely manage it, while slightly weaker models lost control frequently. This context explains why, beyond Prompt Engineering, task-specific fine-tuning (Tuning) for Agent scenarios became a necessary complement for production deployments.

Agent Tuning: Development Workflow and Cost Considerations

A typical development workflow forms a closed loop:

1. Use a powerful model like GPT-4 (open-source or commercial) to validate and fully work through the entire Agent business workflow;
2. Build training data based on the validated Prompts (Prompts and training data are tightly coupled) — automated generation plus manual correction;
3. Use that data to perform Agent Tuning on your own model;
4. Replace the powerful model from step one with the fine-tuned model, completing the loop.

Cost is a real consideration here. The course explicitly notes that Agent Tuning is not necessary for every business scenario. If you can call OpenAI or a similar API directly and the cost is acceptable, there's no reason to train your own model.

Using a 7B model as an example: the training data is approximately 4.7 million tokens, trained for 5 epochs; at Alibaba Cloud A100 pricing, you'd need a four-GPU setup to run it, with a clear time and cost estimate for a full training run. Factor in multiple rounds of iteration and costs grow further. The value of these concrete cost estimates is to help teams make rational decisions — whether to do Tuning is fundamentally a cost-benefit trade-off.

A "token" is the basic unit of cost accounting for LLM training and inference — roughly 4 English characters or 1.5 Chinese characters per token. 4.7 million tokens of training data is roughly equivalent to several thousand complete conversation examples. A training epoch refers to one complete pass through the entire training dataset; more epochs increase task-specific adaptation but also raise the risk of overfitting. The A100 is NVIDIA's flagship GPU for LLM training, with 80GB of VRAM per card; multi-card parallel setups are standard for training models with 7B or more parameters. These cost estimates provide a quantitative baseline for teams evaluating the investment required to build their own Agent capabilities — actual costs will vary with data scale, number of epochs, and cloud provider pricing.