Build an AI Agent Framework from Scratch: A Full Walkthrough of Function Call and MCP

Why Build an Agent Framework from Scratch

Building a simple AI Agent isn't hard these days — frameworks like LangGraph and Swarm make the onboarding experience smooth. But as the creator of this course puts it, the real goal here is demystification: making agents no longer a black box for developers.

The entire course avoids any pre-built framework and writes everything from scratch, deeply dissecting the underlying logic of the "model–tool–memory" triad. The biggest payoff is gaining a clear view of every detail in Function Call, multi-turn dialogue, MCP tool invocation, and other critical mechanisms — free from the opacity that frameworks introduce.

One of the author's industry observations is worth noting: many companies use frameworks like LangGraph or Swarm during the Demo phase, but once a project matures and business requirements become highly specific, teams often gravitate toward building in-house. The reason is simple — custom code is flexible enough to do exactly what you need and nothing more. Frameworks, with all their built-in constraints, often can't satisfy the customization demands of complex production systems.

Four Pain Points of Prompt Engineering

The course opens by addressing a common misconception. Some people believe "the core of large language models is all about prompts" — the author playfully calls such people "gods of the prompt." This view is partially right: aside from training the model directly, prompts are indeed the primary lever we have. But prompts aren't something you can just slap together and expect to work. There's a whole methodology behind them, giving rise to techniques like zero-shot learning, few-shot learning, and Chain-of-Thought (CoT).

Chain-of-Thought (CoT) Background: CoT was formally introduced by Wei et al. from the Google Brain team in a 2022 paper. The key finding was that when prompts include examples of step-by-step reasoning, LLMs perform significantly better on complex reasoning tasks. Subsequent research produced "zero-shot CoT" (triggered simply by adding "Let's think step by step"), "Tree of Thought (ToT)", and other variants. The consensus explanation for why CoT works is that intermediate steps provide the model with extra "computation space," allowing it to decompose complex problems — a design philosophy that aligns closely with the multi-step tool-calling approach used in agents.

Building a system centered solely on prompts exposes four obvious problems:

• Fragility and non-reproducibility: Changing a few words that seem equivalent to humans can produce wildly different model outputs. This unpredictability is fine for demos but unacceptable in production.
• Poor scalability: As the number of tools grows, pure prompt-based solutions scale poorly.
• Heavy burden on users: Offloading the complexity of prompt engineering to end users makes for a terrible experience.
• Statelessness: LLMs have no inherent memory, making multi-turn conversations difficult to handle.

For these reasons, the industry has evolved from pure "prompt engineering" to "Context Engineering" — systematically addressing these issues by introducing short-term memory, RAG, standardized tool output, and more.

"Context Engineering" is a concept that gradually took shape in the industry post-2024, popularized by figures like Andrej Karpathy. Its key distinction from prompt engineering is this: prompt engineering focuses on optimizing the wording of a single input, while context engineering focuses on systematically managing all the information a model can see throughout an entire task — including conversation history, retrieved documents (RAG), tool descriptions, execution results, and so on. This shift reflects a deeper understanding of LLM limitations: a model's ceiling is often determined not by its parameters, but by the quality of the context it receives.

Function Call: Teaching LLMs to Use Tools

At their core, LLMs are "string in, string out" systems. Using them for Q&A is straightforward, but invoking tools requires a mechanism — feeding the model a tool Schema (description) as a string.

Function Calling is the core mechanism for LLMs to interact with external tools. Technically, models are trained with data specifically formatted for tool invocation, enabling them to parse structured tool descriptions and output well-formed invocation instructions. OpenAI standardized this pattern in GPT-3.5/4 in 2023, and it has since become an industry norm. Importantly, Function Call isn't truly "calling" anything — the model itself can't execute code. It simply outputs a structured JSON string; the actual execution is handled by the host program, which then feeds the result back to the model to continue reasoning.

A tool Schema is essentially a structured description of a tool, containing: a description of what the tool does, the name, type (string/number/boolean), and meaning of each parameter, and required flags indicating which parameters are mandatory. The LLM analyzes these descriptions to decide which tool to call and what arguments to pass.

Interestingly, this format must follow a specific convention because it corresponds to the format agreed upon during model training — the tool format at inference time must match the training data for the model to parse it correctly.

Using an LLM to Generate Schemas — Relaxing Annotation Requirements

The author makes a clever engineering optimization here. The traditional approach (used by some frameworks) requires developers to write docstrings in strict Google or NumPy style; non-compliant comments cause parsing failures.

The author's approach instead takes the function's docstring (function.__doc__) along with the target Schema structure, injects them into a prompt, and lets an LLM handle the "natural language comment → structured JSON" conversion. This means developers can write fairly casual docstrings — as long as the description is clear, it works.

Since LLM outputs are non-deterministic, the author implements a retry loop of up to three attempts — experience shows three tries is almost always sufficient, and if all three fail, an error is raised.

MCP Protocol: Mirroring Remote Services as Local Tools

Local tools are handled, but many tools are actually deployed on remote MCP servers and can't be called locally. The author's solution is "remote service mirroring" — directly mirroring remote MCP services as local tool functions.

MCP (Model Context Protocol) was released by Anthropic in November 2024 to address the fragmented integration between AI models and external data sources and tools. Before MCP, every AI application had to develop a custom adapter for each external service, creating an "N×M" integration complexity. MCP simplifies this to an "N+M" model by defining a standardized server-client communication protocol — service providers implement the MCP server once, and AI applications implement the MCP client once, and they can all interoperate. Hundreds of popular services (GitHub, Slack, various databases, etc.) now offer official MCP support, and it is rapidly becoming the de facto standard for AI tool ecosystems.

In the demo, the MCP service provides two tools: one that queries weather (pulling real data directly from the Alibaba Cloud website), and one that converts currencies (randomly generated for demonstration purposes). Using get_tools, the remote tool names and descriptions are retrieved, assembled into local function bodies, and forwarded to the remote MCP when called. These mirrored functions don't even need to be written to disk — they're loaded directly into memory and ready to use.

This is where the core value of MCP's standardized protocol shines: without MCP, every developer builds their own service interface, requiring you to write custom mirroring logic for each one — a massive overhead. With MCP as the standard, you implement the mirroring logic once, and any compliant service can be converted into a local tool function. The architecture becomes instantly clean.

In the main function, local tools (Baidu search, get current time, addition) and MCP tools are treated uniformly — "use MCP function tools just like local tools" — keeping the entire architecture consistent and clean.

Dual-Model Dispatch Architecture and the Thinking Loop

The agent's overall execution flow breaks down into two major steps: user input → think and decide whether to call a tool → call tool and think again → repeat until no more tools are needed → produce final answer. Since a single request may involve multiple tool calls (e.g., querying weather for both Beijing and Shanghai simultaneously), a loop mechanism is designed here. This "think-act loop" has a classic academic formalization — ReAct (Reasoning + Acting), introduced by Yao et al. in 2022, and one of the most influential agent design paradigms today.

Dispatch Model vs. Chat Model

The author introduces a dual-model architecture:

• Dispatch model: A more capable model that decides whether to call a tool and determines the parameters. This step demands stronger reasoning ability.
• Chat model: If the query is determined to be casual conversation, it's handled by a lighter model, effectively reducing cost.

The system maintains a state machine to manage flow transitions: the initial state is chat; when a tool is needed, it transitions to tool state to execute the call; after execution, the result is passed through an LLM for a summarized description before being returned to the user. The state machine is a classic computer science model for describing system state transitions. Its core value in this architecture is traceability: when agent behavior goes wrong, developers can pinpoint exactly which node the problem occurred at by reviewing state transition logs — rather than staring at an opaque black box. This is the engineering philosophy of "de-black-boxing" in action.

Temporary Messages Don't Enter the Context

A key engineering detail: during the thinking phase, tool Schemas and guiding prompts (e.g., "based on the previous result, execute the next step") are temporarily injected — but these guiding messages do not enter the long-term context.

The reason: an overly long context degrades model performance, so only the most effective information should reach the model. Tool Schemas are only included during the dispatch phase, not during regular conversation, which shortens input length and allows the model to process more high-signal information.

Short-Term Memory: Giving the AI Agent Conversational Context

Memory is implemented via a MemoryManager that stores all multi-turn conversation history, forming what's known as short-term memory.

The demo illustrates this clearly: ask a raw LLM "who am I?" and it doesn't know; tell it "I'm Pitan's Dad" and ask again, and it remembers and answers correctly. For more sophisticated setups, memory can be persisted to Redis or a database, or even enhanced with RAG (Retrieval-Augmented Generation) for long-term memory management.

RAG was proposed by Meta AI in 2020, and in an agent memory architecture it serves the role of "long-term memory": historical conversations, user preferences, and domain knowledge are vectorized and stored in vector databases (such as Pinecone, Chroma, or Milvus), with the most relevant snippets retrieved and injected into context at each conversation turn. Compared to the "short-term memory" approach of cramming the entire conversation history into the context window, RAG can handle histories far exceeding the context length limit — but at the cost of introducing retrieval accuracy uncertainty. Both memory mechanisms are typically used together in production-grade agents, collectively forming a complete memory system.

Core Conclusion: Agents Are Fundamentally an Engineering Problem

After walking through the entire from-scratch implementation, the author offers a thought-provoking conclusion: agents have relatively little to do with AI algorithms per se — they are primarily an exercise in engineering encapsulation.

This explains why understanding the underlying principles matters so much. Once you've fully deconstructed Function Call, MCP mirroring, state machine loops, context management, and memory storage — the "engineering building blocks" — you can freely combine them to meet any specific business requirement, unconstrained by any framework's limitations. That is the ultimate goal of this course: demystification.