Hung-yi Lee's 2026 Course Notes: Deep Dive into the Three Core Capabilities of AI Agents

What Is an AI Agent? Professor Hung-yi Lee's Definition

In Professor Hung-yi Lee's latest 2026 AI Agent course series, he begins by clearly defining AI Agents: Humans don't provide explicit behavioral instructions or step-by-step guidance — they simply give the AI a goal, and the AI figures out how to achieve it on its own.

This is fundamentally different from how we typically use AI. Normally, we give AI a specific instruction (e.g., "translate this paragraph"), and the AI executes it accordingly. But AI Agents need to handle complex, multi-step tasks — for example, given a research topic, an AI Agent must formulate hypotheses, design experiments, run them, analyze results, and even go back to revise hypotheses when results don't meet expectations.

The workflow of an AI Agent can be summarized as: Receive goal → Observe the environment (Observation) → Decide on an action (Action) → Affect the environment → Obtain new observations → Repeat the cycle until the goal is achieved. AlphaGo is a classic AI Agent example: the goal is to win the game, the observation is the board state, and the action is choosing where to place a stone.

From Reinforcement Learning to LLM-Driven AI Agents

In the past, building AI Agents primarily relied on Reinforcement Learning (RL) algorithms, but this approach had clear limitations: you needed to train a dedicated model for every single task. AlphaGo can play Go but not chess — playing chess requires a separately trained model.

The core reason AI Agents are surging in popularity today is a new idea: Can we directly use Large Language Models (LLMs) as AI Agents?

Using LLMs to drive AI Agents offers two major advantages:

1. Near-infinite action space: AlphaGo can only choose among positions on a 19×19 board, while LLMs can generate virtually unlimited types of output, freeing Agents from constrained action spaces.
2. No need to define rewards: RL requires manually defining reward functions (why is a compilation error -1 and not -17.7?), while LLMs can directly read error logs and obtain feedback far richer than a single numerical value.

From the LLM's perspective, when it acts as an AI Agent, it's doing nothing fundamentally different — it's still just doing next-token prediction. AI Agents aren't a new technology for LLMs; they're a new way of applying LLMs.

Real-World Applications of AI Agents

Virtual Villages and Game NPCs

As early as 2023, researchers used language models to run NPCs in a virtual village, where each NPC had human-defined goals (throwing a party, studying for exams, etc.) and made decisions based on text-described environmental information. Later, someone even replaced all NPCs in Minecraft with AI-driven ones, and reportedly these AIs organized their own financial systems and governments.

AI Operating Computers

A more practical application is letting AI operate computers just like humans. Notable products include Claude's Computer Use and ChatGPT's Operator. These Agents observe the computer screen and take actions by pressing keys or clicking the mouse.

AI Training AI

Using AI to train other AI models is another important application. The process works as follows: set a goal (e.g., surpass a certain baseline) → LLM writes the training program → obtains model accuracy → rewrites the program based on results, repeating the cycle. Well-known frameworks include AIDE and Auto Cargo.

Core Capability 1: Adjusting Behavior Based on Experience (Memory Mechanism)

Why Is Memory Needed?

AI Agents need to adjust behavior based on past experience, but you can't have them review their entire "life story" before every decision — it's like people with hyperthymesia, who are overwhelmed by trivial memories and struggle with abstract thinking.

The solution is to introduce a Memory (long-term memory) mechanism:

• Read module: Retrieves experiences relevant to the current problem from Memory (essentially RAG technology)
• Write module: Decides what information is worth recording and what can be "let go"
• Reflection module: Performs high-level abstraction and organization of memories to generate new insights

Key Findings from StreamBench

In the StreamBench benchmark, a critical finding emerged: Negative feedback is essentially unhelpful — positive examples are far more effective than negative ones. Rather than telling a language model "what not to do," it's better to tell it "what to do" — consistent with previous research showing that saying "write it shorter" works better than "don't write too long."

Core Capability 2: Using Tools (Function Calling)

The General Approach to Tool Use

What we call "tools" are simply functions from the language model's perspective. The general method for using tools is straightforward: Tell the model in the System Prompt what tools are available and how to use them, and the model will output a calling instruction when needed.

The key workflow: Model outputs tool-calling text → Developer intercepts and executes the actual function → Results are injected back into the model's context → Model continues generating the final response.

What Happens When There Are Too Many Tools?

When available tools number in the hundreds or thousands, you can't have the model read through every manual before acting. The solution is similar to Memory's RAG mechanism: store tool descriptions in a toolkit and use a retrieval module to select appropriate tools based on the current state.

Taking it a step further, language models can even create their own tools — writing a function, validating it works, and then storing it in the toolkit for future use.

Can AI Be "Tricked" by Tools?

Research has found that language models have some ability to judge tool outputs (they won't question 100°F but will question 10,000°F), though the risk of being misled still exists. Factors that influence whether AI trusts external information include:

• The degree of deviation between external knowledge and the model's internal beliefs (greater deviation = less trust)
• The model's confidence level in its own answer (higher confidence = harder to sway)
• The publication date of the article (more likely to trust newer articles)
• Whether the article was written by AI (more likely to trust fellow AI)

Core Capability 3: Planning Ability

How Good Is Current Model Planning?

In PlanBench's "Mystery Blocksworld" test (which uses unconventional rules to prevent models from simply copying training data), 2023 models failed almost entirely. However, the emergence of reasoning models like O1 brought significant improvement.

In travel planning tasks, early 2024 models had success rates near 0%, but with tool assistance (such as constraint solvers), GPT-4 and Claude 3 achieved over 90% accuracy.

Tree Search and Mental Simulation

One method to enhance planning ability is to have the model perform Tree Search — trying different paths, evaluating success probability, and pruning hopeless branches. But in reality, many actions are irreversible (like a pizza order that's already been placed), so a more practical approach is to have all searching happen in a "mental theater" — the model imagines execution results, finds the optimal path in its mind, and only then actually executes.

Interestingly, models with reasoning capabilities (such as DeepSeek R1), when acting as AI Agents, naturally perform similar planning within their Chain of Thought — trying different possibilities on their own, self-verifying, and essentially playing the role of a World Model.

Overthinking: The Danger of Thinking Too Much

Recent research titled "The Danger of Overthinking" points out that while models capable of mental simulation perform better overall, they suffer from overthinking: some models endlessly deliberate about what will happen if they click a button but never act, and some even give up before trying. How to prevent models from "dying of overthinking" is an important research direction for the future.

Summary: The Full Technical Landscape of AI Agents

Professor Hung-yi Lee's lecture dissects the core capabilities of AI Agents across three dimensions:

• Adjusting behavior based on experience: Memory + RAG retrieval mechanism
• Using tools: Function Calling + intelligent tool selection
• Planning ability: Tree Search + mental simulation

Current AI Agents don't rely on entirely new technologies — they fully leverage the existing next-token prediction capabilities of LLMs and achieve autonomous completion of complex tasks through clever engineering design. This course provides a clear and systematic framework for understanding and building AI Agents, while also revealing the boundaries of current technology and future research directions.