Building an AI Agent Framework from Scratch: Deep Dive into Tool Calling, Memory, and Dual-Model Scheduling

It All Starts with Prompts — and Their Four Fatal Limitations

A colleague once jokingly called the large language model's core a "Greek librarian of prompts." That's partly right, partly wrong.

Without fine-tuning, prompts are indeed the primary lever we have over LLM behavior, giving rise to zero-shot learning, few-shot learning, and Chain-of-Thought (CoT). Zero-shot learning means describing a task in natural language with no examples; few-shot learning includes 1–5 input-output examples in the prompt to demonstrate the expected format — neither requires updating model weights. Chain-of-Thought (CoT), proposed by the Google Brain team in 2022, guides models to emit intermediate reasoning steps before a final answer, significantly improving performance on math and symbolic reasoning tasks, especially for models over 100B parameters.

But building a system entirely on prompts exposes four critical problems:

Fragility and non-reproducibility: Trivial wording changes can produce wildly different outputs. Fine for demos, dangerous in production.
Poor scalability: As more tools are added, a pure prompt-based approach becomes impossible to maintain.
Heavy user burden: Offloading prompt engineering complexity to users simply isn't realistic.
Statelessness: LLMs have no native memory, making multi-turn conversations difficult.

This is why the industry has evolved from "prompt engineering" to "context engineering" — a concept championed by Anthropic and others since 2024. Context engineering isn't just about what to say, but about carefully managing what information goes in, when, and in what structure.

This approach has solid academic backing. Stanford's 2023 "Lost in the Middle" study revealed a structural bias in LLMs: models recall and use information at the beginning and end of inputs far better than information buried in the middle. Placing critical instructions in the middle of a long context is likely to be ignored. Key constraints should go at the top; the latest tool results should be near the end. The study also showed that noise (irrelevant text) degrades reasoning accuracy non-linearly — justifying aggressive context trimming in system design.

This implementation covers short-term memory, standardized tool output, and more — but intentionally omits RAG (Retrieval-Augmented Generation). RAG, introduced by Meta AI in 2020, retrieves relevant passages from a vector database before generation to compensate for model knowledge cutoffs. Vector databases (Faiss, Chroma, Weaviate, Pinecone) index high-dimensional embeddings for fast semantic search. This module is left as an exercise for the reader.

Querying weather directly from Alibaba Cloud's official website

The Essence of Function Call: Teaching LLMs to Understand Tool Schemas

At their core, LLMs are "string in, string out" systems. For Q&A that's fine, but invoking external tools requires a bridge — the tool schema.

Function Calling was first introduced by OpenAI in 2023 and quickly became an industry standard. Instead of returning natural language, the model outputs a structured JSON instruction specifying which function to call and with what parameters.

This has since become a cross-vendor standard: Google Gemini's Function Calling, Anthropic Claude's Tool Use, and open-source models like Mistral and Qwen all support JSON Schema-based tool descriptions. JSON Schema itself dates to 2009 — a meta-description language for defining the structure of JSON data (types, required fields, enums, nested objects), widely used in API documentation (OpenAPI/Swagger), contract validation, and config file verification. Function Calling borrows this format so that tool invocation instructions are machine-parseable. A single tool schema can, in theory, be reused across different models.

Schema Structure Design

The tool schema is designed as a layered structure: the top level describes the tool's purpose; below that are parameter definitions. Each parameter specifies:

Type: string, number, or boolean
Description: a detailed explanation of the parameter
required: whether the parameter is mandatory

This format isn't arbitrary — it should closely match the tool format the model was trained on, so the model can accurately parse it and generate correct parameters at inference time.

Auto-Generating Schemas with an LLM

This is one of the cleverest design choices in this implementation. Conventional approaches (seen in many tutorials) require developers to write function docstrings in strict Google or NumPy format, then parse them with substantial code — rigid, and brittle if the format deviates.

The solution here: let an LLM convert docstrings into schemas. The flow:

Define the target schema structure using BaseModel
Inject the function docstring (function.__doc__) into a prompt
Ask the LLM to output a structured JSON schema with no extra content
Parse and return the JSON

To account for LLM non-determinism, a retry loop of three attempts is built in — one failure doesn't mean the next will fail too. Three failures in a row suggests the conversion is genuinely impossible, at which point an error is raised. This means developers can write fairly casual docstrings, as long as the description is clear enough to be auto-parsed.

Tool schema type definitions

MCP Services: Mirroring Remote Tools as Local Functions

Local tools are solved — but many tools are deployed on remote MCP (Model Context Protocol) servers that can't be called directly.

MCP is an open protocol released by Anthropic in late 2024 to address the fragmented integration of AI models with external tools and data sources. Analogous to USB-C unifying hardware connections, MCP provides a unified communication standard for AI tool invocation — covering tool discovery, call requests, and result returns — so tools from different providers can be consumed consistently.

MCP's deeper significance lies in its tool discovery mechanism: instead of hardcoding a tool list at deploy time, an Agent can dynamically query an MCP server at runtime for its available tools and their schemas. New tools come online without modifying the Agent's core code. This mirrors service discovery patterns in microservices (Consul, Eureka) — providers register their capabilities, consumers query on demand — and is a foundational concept for enterprise-grade AI tool platforms.

How Mirroring Works

The approach here is remote service mirroring — turning remote MCP services directly into local tool functions.

get_tools fetches the remote function names and descriptions, assembles a function body, and at execution time calls the remote MCP endpoint. These mirrored local functions don't need to be written to disk — they're loaded directly into memory.

In the demo, a remote MCP service exposes two tools: weather lookup (using real data from Alibaba Cloud) and currency exchange (using random values for demonstration). After mirroring, they appear as two ordinary local functions.

The Core Value of Standardized Protocols Like MCP

This is where the value of a unified standard becomes clear. Without one, every service has a different API — mirroring each requires custom adaptation, and the cost compounds quickly.

With MCP's unified standard, any service can be converted to a local tool function with the same logic. Local tools (Baidu search, time lookup, arithmetic) and MCP-mirrored tools can then be managed uniformly — the architecture stays clean.

Unifying remote and local tools

Dual-Model Scheduling: Dispatch Model vs. Chat Model

The Agent's execution loop can be summarized as: user input → think (call tools?) → call tools → get results → think again → … → final answer with no further tool calls.

For example, "check the weather in Beijing and Shanghai" triggers two tool call rounds: first Beijing, then Shanghai, then a summary.

Why Two Models?

A key improvement over the standard loop: a dual-model architecture:

Dispatch model: A more capable model that decides whether to call tools and generates the correct parameters — this step demands stronger reasoning.
Chat model: A lighter, cheaper model that handles casual conversation only.

Dual- or multi-model collaboration is a common cost-optimization strategy in enterprise AI systems, also known as LLM Routing. Large reasoning models (GPT-4o, Claude 3.5 Sonnet) can cost 10–20× more per API call than lightweight models (GPT-4o-mini, Claude Haiku). The challenge is classifier accuracy — too aggressive and complex queries get misrouted to weak models; too conservative and you lose the savings. Projects like RouteLLM and Martian, as well as cascade approaches (try a small model first, escalate if confidence is low), can reduce API costs by 30–70% while preserving user experience. This implementation uses a state machine-driven rule router — simple, reliable, and a pragmatic starting point.

A Finite State Machine (FSM) manages switching between the two modes (chat state vs. tool state). FSMs are a classic formal model in computer science, composed of a finite set of states, transition conditions, and corresponding actions — widely used in compiler lexing, network protocols, and game AI behavior trees. In an Agent system, the FSM decomposes complex execution flows into explicit states with defined transition triggers, avoiding scattered conditional logic. LangGraph uses directed graphs as a visual FSM representation — nodes are states, edges are transitions — enabling debuggable, replayable Agent execution.

State label for tool-calling rounds

The Engineering Wisdom of Context Trimming

Here's an easy-to-miss but critical detail. During the "think" phase (deciding whether to call a tool), the tool schemas and a temporary guiding message are injected to steer the model — but these never enter the permanent context.

The reason: a longer context degrades LLM performance. Tool schemas are only needed at the dispatch step; carrying them into the full conversation would inflate input length and dilute information density. This directly echoes the "Lost in the Middle" findings — actively controlling context length and stripping phase-specific noise is a mandatory engineering practice, not an optional optimization.

This also explains a common industry pattern: companies build demos with LangGraph or Swarm, then trend toward in-house implementations as projects deepen. The reason is flexibility — you write exactly what you need and remove exactly what you don't. Off-the-shelf frameworks become obstacles when business requirements are sufficiently unique.

Short-Term Memory: Where Does an Agent's "Memory" Come From?

The demo includes an instructive illustration: ask the model "who am I?" out of the box and it has no idea; tell it "I'm Pitan's dad" and ask again, and it answers correctly. This is short-term memory in action.

The implementation relies on a Memory Manager — storing the full multi-turn conversation history and injecting it into the context as short-term memory.

The stateless nature of LLMs means memory systems must be built externally. The industry typically identifies four memory layers:

Buffer Memory: Retains the full conversation history. Zero information loss, but token cost grows linearly with turns. This implementation's Memory Manager falls here — the most intuitive approach for understanding memory fundamentals.
Sliding Window Memory: Keeps only the last N turns. Balances token efficiency with short-term coherence; suitable when long histories aren't needed.
Summary Memory: Periodically compresses history into a summary using the model. Uses fewer tokens to maintain historical context, but risks lossy compression of critical details.
Vector Memory: Embeds conversation history into a vector database (Faiss, Chroma, Weaviate) for semantic retrieval on demand. The mainstream approach for "long-term memory" and a direct application of RAG in the memory context.

For more robust implementations, you could:

Store history in Redis for more durable caching
Persist to a database for long-term memory

The variations are many, but the core idea is consistent: use engineering to compensate for the LLM's innate statelessness. Every LLM inference is an independent stateless process — it doesn't "remember" the previous turn. Memory is fundamentally a matter of serializing conversation history back into the input to simulate continuity.

Conclusion: An Agent Is Ultimately Good Engineering Encapsulation

The greatest value of building an Agent framework from scratch is demystification. Once you've personally implemented Function Call schema parsing, MCP remote mirroring, dual-model state scheduling, and memory management, you'll arrive at the same conclusion:

Agents have relatively little to do with AI algorithms themselves — they are primarily well-engineered encapsulations of a series of components.

For developers who are learning or in a transitional phase: rather than relying entirely on black-box frameworks, implement each layer yourself at least once. Understanding the underlying mechanics will serve you well — whether tackling interview questions about the four limitations of prompt engineering, or doing deep customization in real production systems.