Building an AI Agent from Scratch: Implementing Core Conversational Agent Features in Python

Introduction

With AI Agents becoming a major tech trend, many developers want to get hands-on but don't know where to start. Based on a practical tutorial, this article walks you through the two core concepts of an Agent—large language models and prompts—and guides you through a complete Python implementation of a basic conversational agent. Mastering this step is the starting point for building more complex Agents.

Understanding the Two Core Components of an Agent: Large Language Models and Prompts

An AI Agent is an AI system capable of perceiving its environment, making autonomous decisions, and executing tasks. Unlike traditional single-turn Q&A AI, an Agent has goal-oriented behavior, continuous interaction capabilities, and tool-calling abilities, enabling it to decompose complex tasks into multiple steps and complete them one by one.

From an architectural perspective, an Agent is essentially a closed-loop "perception-decision-action" system: the perception layer receives external inputs (text, images, sensor data, etc.); the decision layer, handled by a large language model, is responsible for understanding intent and planning steps; and the action layer interacts with the external world through Function Calling. This architecture aligns closely with the "agent-environment" interaction paradigm in reinforcement learning—in classical RL, an agent observes the environment state (Observation), selects an action (Action), receives a reward signal (Reward) from the environment, transitions to a new state, and updates its policy accordingly. The core innovation of LLM-based Agents is replacing the traditional Policy Network with a pretrained language model: while traditional policy networks require millions of interaction samples in a specific environment to learn a single task, language models leverage world knowledge accumulated during pretraining to generalize to new tasks without extensive environment interaction. This lowers the development barrier from "requiring specialized RL engineers" to "developers who can write prompts." The current Agent boom is primarily driven by breakthroughs in LLM reasoning capabilities—models like GPT-4, Claude, and DeepSeek now possess sufficient instruction comprehension and logical reasoning abilities to make Agents practical rather than merely conceptual.

To build an agent, you must first understand its two most important components: large language models and prompts.

The large language model is the core of the entire agent—think of it as a "super brain." From a technical perspective, a Large Language Model (LLM) is a neural network model based on the Transformer architecture, trained on massive amounts of text data. The Transformer architecture was introduced by Google in the 2017 paper Attention Is All You Need, with its core innovation being the "Self-Attention" mechanism.

The self-attention mechanism works as follows: for each word in the input sequence, the model maps it into three vectors—Query (representing "what I want to attend to"), Key (representing "what information I can provide"), and Value (representing "the actual information I carry"). By computing the dot product of the Query with all Keys, applying Softmax normalization to obtain attention weight distributions, and then computing a weighted sum of the Value vectors using these weights, the model obtains a new representation for that word that incorporates global contextual information. This mechanism allows the model to dynamically determine whether "bank" refers to a "financial institution" or a "riverbank" based on context, thereby capturing long-range semantic dependencies.

It's worth understanding that the "Multi-Head" variant of self-attention is standard in modern large models: by projecting Query/Key/Value into multiple different subspaces and computing attention in parallel, the model can simultaneously capture dependencies between words from multiple semantic dimensions (such as syntactic relationships, semantic similarity, coreference, etc.), and then concatenate and fuse the outputs from all heads. This design is similar to how multiple convolutional kernels in CNNs extract features from different perspectives, giving the model richer expressive power. Compared to the previously dominant RNN/LSTM architectures—which need to sequentially pass hidden states and where early information in long sequences easily suffers from vanishing gradients during backpropagation—Transformer's self-attention allows direct interaction between any two positions and enables parallel computation across all positions, making training on ultra-large datasets feasible. Modern large language models are built on the Decoder-only variant of the Transformer, using Causal Attention to ensure each position can only attend to preceding Tokens, with parameter counts ranging from billions to trillions.

The core mechanism of large language models is "next token prediction"—the model learns statistical patterns of language and predicts the most likely next word given a context. Modern models like DeepSeek and the GPT series typically undergo additional RLHF (Reinforcement Learning from Human Feedback) alignment training. RLHF addresses the gap between "capability" and "intent alignment"—a pretrained model may have powerful language capabilities but not know how to use them in ways beneficial to humans. The RLHF process has three stages: first, supervised fine-tuning (SFT) of the pretrained model; second, training a Reward Model by having human annotators rank multiple model outputs to learn human preferences; finally, using reinforcement learning algorithms to optimize the language model using the reward model's scores as signals. In recent years, DPO (Direct Preference Optimization) has been widely adopted as a simplified alternative to RLHF—it directly optimizes the model from preference data without needing a separate reward model, reducing training complexity. OpenAI's InstructGPT paper demonstrated that a 1.3B parameter model trained with RLHF even outperformed an unaligned 175B model in human evaluation, showing that alignment training improves practicality far more than simply scaling parameters. When users converse with the large model, it generates responses based on input. In this interaction process, what users send to the AI is essentially a prompt.

Two Types of Prompts

Prompts are the instructions or inputs sent to an AI model—they determine how the AI understands your intent and responds. From the model's internal mechanism perspective, prompt quality directly affects the model's "attention allocation"—well-crafted prompts can activate relevant knowledge and reasoning patterns learned during pretraining, while vague or ambiguous prompts may scatter attention across irrelevant information, ultimately determining output quality. Based on their function, prompts are divided into two types:

• System Prompt: Sets the AI's base persona, behavioral norms, and global constraints. It's like establishing "ground rules" for the AI that remain in effect throughout the entire conversation. From a technical implementation standpoint, the system prompt is placed at the beginning of the messages list with role: system in every API call, and the model is conditioned on it when generating every single Token. Note that system prompts are not an absolutely unbreakable "firewall"—"Prompt Injection" attacks are an important research topic in current Agent security. Attackers may craft user inputs that induce the model to ignore system prompt constraints, which is why production environments need additional security layers.
• User Prompt: The specific content of each user question—the actual input for each conversation turn.

Here's an intuitive example: when you open the DeepSeek website and type "hello," that "hello" is a user prompt. The underlying large model receives it and generates a response. Now that you understand this basic flow, let's reproduce it in code.

Environment Setup and Module Imports

The first step is setting up the development environment. Create a local virtual environment in VS Code and create a new code file.

Next, you need to import two key third-party modules:

• OpenAI: For calling LLM APIs compatible with the OpenAI interface. Notably, the OpenAI API format has become the de facto standard for LLM interfaces—DeepSeek, Moonshot, Zhipu AI, and many other domestic and international model providers offer APIs compatible with the OpenAI format. The OpenAI API format became an industry standard for multiple reasons: OpenAI pioneered commercial LLM APIs, and its interface design has been validated by a large developer community; the messages list + role design intuitively maps the natural structure of multi-turn conversations; and extension capabilities like Function Calling and Streaming are well-designed and easy to implement. For model providers, OpenAI format compatibility means directly leveraging the vast OpenAI ecosystem toolchain (frameworks like LangChain, LlamaIndex, etc.), significantly lowering migration barriers for developers and creating a positive ecosystem flywheel effect. This means developers only need to learn one set of interface specifications to seamlessly switch between different underlying models, greatly reducing migration costs;
• python-dotenv: For reading API keys and other configuration from local .env files. This practice follows the "separation of configuration and code" engineering principle (the third factor in the Twelve-Factor App methodology), preventing sensitive information from appearing in code repositories and avoiding key leakage from accidental commits to public platforms like GitHub.

Install these two modules via pip install in the virtual environment's terminal. This step forms the foundation for all subsequent development.

Deep Dive: The Engineering Value of Python Virtual Environments Python virtual environments (venv) are the standard practice for isolating project dependencies. Different projects may depend on different versions of the same library (e.g., one project needs openai==0.x while another needs openai>=1.0)—sharing a global environment would create version conflicts. Virtual environments create independent Python interpreters and package directories for each project, completely eliminating dependency pollution. In team collaboration, they're typically paired with requirements.txt or pyproject.toml to lock dependency versions, ensuring complete consistency across different developers, CI/CD pipelines, and production environments—a concrete manifestation of the "reproducible builds" engineering principle.

Calling the DeepSeek API for Basic Conversations

With the environment and modules ready, the next step is implementing the backend conversation interaction. The key technique is: directly reference the DeepSeek official API documentation.

DeepSeek's API documentation provides standard code examples for calling the conversation interface. Copy it into your file and make minor modifications to get a basic conversation program.

Securely Managing API Keys

Secure API key management is an important engineering practice. Hardcoding API keys directly in code is neither safe nor professional. The recommended approach is:

1. Create a .env file;
2. Enter the API Key obtained from the DeepSeek website;
3. Read the key from local environment variables in your code.

The python-dotenv library injects sensitive configurations into environment variables by reading the .env file. In team collaboration scenarios, the .env file is typically added to .gitignore, with a companion .env.example template file for team members to reference configuration items. In production environments, key management is further upgraded to professional secret management services like AWS Secrets Manager or HashiCorp Vault, which provide enterprise-grade security capabilities such as Key Rotation, access auditing, and fine-grained permission control (based on IAM policies)—key rotation is particularly important as it ensures that even if a key is leaked, its validity is limited to a short time window, significantly reducing security risk.

Deep Dive: Real Risks of API Key Leakage API key leakage has serious consequences: attackers can use leaked keys to make unlimited calls to paid APIs, generating enormous bills (known in the industry as "Bill Hijacking"); in enterprise scenarios, it may also lead to data breaches or service disruptions. GitHub's Secret Scanning feature automatically scans public repositories for known key formats and issues alerts, but this is still a reactive measure. Best practice is to block commits containing keys locally before they're pushed using tools like git-secrets or pre-commit, eliminating risk at the source rather than relying on platform-side post-detection.

The process for obtaining an API Key is straightforward: go to the API documentation page, create an API Key, enter a username, and click create. This way the code itself doesn't expose sensitive information, and keys are managed centrally in the .env file—both secure and maintainable.

Implementing Multi-Turn Conversations: Making the Agent Interactive

The initial code has an obvious limitation: each run only sends one conversation, and the content is hardcoded. To turn the program into a truly interactive Agent, you need to introduce a loop mechanism.

Adding a While Loop for Continuous Interaction

Here's the specific approach:

1. Wrap the conversation logic in a while loop so the program can repeatedly receive input;
2. Use the input() function to dynamically capture user input each time;
3. Set an exit condition—terminate the loop when the user types "exit."

After this modification, the program can continuously receive user questions and return responses, creating a true multi-turn interactive experience.

Organizing the Messages Structure

In API calls, the messages parameter carries the conversation content. The OpenAI API's messages parameter uses a role + content list structure supporting three roles: system, user, and assistant. This design simulates real conversation context stacking—each turn is appended to the list, and the model "sees" the complete conversation history when generating responses. This is the fundamental reason multi-turn conversations maintain contextual coherence. Messages primarily contain two parts:

• System Prompt: The initial setup, such as "You are an assistant";
• User Prompt: Replace hardcoded content with user_input—the user's actual input each time.

It's important to note that as conversation turns increase, the messages list grows longer and may eventually exceed the model's Context Window limit. The context window is the maximum number of Tokens a large language model can process at once—Tokens are the basic units for model text processing, created by the Tokenizer splitting raw text, and are not equivalent to characters: Chinese typically maps 1-2 characters to 1 Token, while English maps roughly 4 characters to 1 Token, which is why Chinese text of equivalent content often consumes more Tokens than English. Early GPT-3 had a context window of only 4K Tokens, while current mainstream models have expanded to 128K or even longer (e.g., the Claude 3 series supports 200K Tokens, equivalent to approximately 150,000 Chinese characters).

Context window expansion comes at a cost: since self-attention's computational complexity is quadratic with sequence length (O(n²)), longer context windows mean higher computational overhead and inference latency—this is why even when models support ultra-long contexts, their use still requires careful trade-offs in practice.

Deep Dive: Tokenizer Design Principles Modern large models predominantly use BPE (Byte Pair Encoding) or SentencePiece sub-word tokenization algorithms. BPE's core approach starts at the character level, statistically identifies the most frequent character pairs in the training corpus, merges them into new sub-words, and iterates until reaching a preset vocabulary size (typically 30,000 to 100,000). This design balances vocabulary coverage with sequence length efficiency: high-frequency words are encoded as single Tokens (e.g., "the"), while low-frequency or novel words are split into multiple sub-word Tokens (e.g., "tokenization" → "token" + "ization"), achieving lossless encoding of arbitrary text while avoiding out-of-vocabulary (OOV) problems. Different models use different tokenizers, so the same text may have different Token counts—something to be particularly aware of when estimating API call costs.

When conversation history exceeds the context window, the model can no longer "see" early conversation content, resulting in context loss. This has spawned multiple memory management strategies:

• Sliding Window (retaining the most recent N turns): Simple to implement but loses important early information;
• Summary Compression (using the model to compress conversation history into a summary injected into the system prompt): Preserves key information but the summarization process itself consumes Tokens and may introduce information loss;
• Vector Database Retrieval (embedding conversation history as vectors for storage, retrieving the most relevant historical segments before each turn): This is a typical application of RAG (Retrieval-Augmented Generation) in Agent memory management—text is converted into high-dimensional vectors via an Embedding Model, stored in vector databases like Pinecone, Chroma, or Milvus, and retrieved by computing cosine similarity to find semantically relevant historical segments, achieving "on-demand recall" rather than "full memory," balancing Token efficiency with information completeness.

These memory management strategies are the core problems that subsequent "memory management" modules need to solve, and represent a key capability difference distinguishing simple chatbots from truly intelligent Agents.

After completing these settings and running the program, type "hello" to see the Agent's response. To make the output clearer, you can display the response content in a bordered format.

Shaping AI Persona with System Prompts

Once the program is working, the final step is experiencing the power of system prompts. System prompt engineering is one of the core competitive advantages in current AI application development. In commercial products, system prompts often encompass multiple dimensions including role definition, capability boundaries, output format specifications, safety restrictions, and business knowledge injection, potentially reaching thousands of words—for example, GitHub Copilot's system prompt injects code style guidelines and repository context; customer service bot system prompts include product knowledge base summaries and standard response scripts; and financial application system prompts include strict compliance disclaimers and risk disclosure requirements.

Prompt Engineering has evolved into an independent technical discipline encompassing multiple core techniques:

Chain-of-Thought (CoT) guides the model to explicitly output intermediate reasoning steps by adding "let's think step by step" to the prompt. Its core principle leverages the model's autoregressive nature—when generating text, previously generated content becomes a condition for subsequent generation. Having the model first output correct intermediate reasoning steps provides stronger probabilistic constraints for final answer generation, effectively decomposing complex single-step reasoning into multiple easier sub-steps. Google research shows CoT can improve mathematical reasoning accuracy by over 40%, which is also the theoretical foundation for why reasoning models like DeepSeek-R1 output extensive "thinking processes."

Few-shot Learning leverages the In-Context Learning capability of large models—the model can infer task patterns from 2-5 examples in the prompt and adapt to specific tasks without gradient updates. This capability is considered one of the core "emergent" abilities that appear when model parameters exceed a certain threshold—small models typically lack this ability. The counterpart is "Zero-shot Learning," where no examples are provided and the task is described purely through natural language—modern large models can handle many general tasks in zero-shot settings, but for tasks with strict formatting or domain-specific requirements, few-shot prompting still significantly improves output quality.

Additionally, Structured Output Prompting (requiring the model to return in JSON format for easy programmatic parsing), Self-Consistency (sampling the same question multiple times and taking the majority answer to improve accuracy), and other techniques are widely applied in production Agent development.

Deep Dive: Emergent Abilities and Scaling Laws In large model research, "Emergent Abilities" refer to capabilities that suddenly manifest when model parameters exceed a certain critical point—capabilities that are virtually nonexistent in smaller models, such as arithmetic reasoning, code generation, and multi-step logical reasoning. This phenomenon is closely related to "Scaling Laws"—OpenAI's research shows power-law relationships between model performance and parameter count, training data volume, and compute, enabling predictable improvement. However, the abrupt nature of emergent abilities breaks simple linear extrapolation: for certain capabilities, small models perform near random chance while large models' accuracy can leap to near-perfect. This pattern profoundly influences industry judgments about large model R&D roadmaps and is one of the core drivers behind major AI labs' continued pursuit of larger-scale training.

Here's a fun demonstration: set the system prompt to "act as a robot and make a 'beep—' sound after every response." After rerunning the program and typing "hello," every response from the AI ends with "beep," vividly presenting the robot persona.

This small experiment intuitively demonstrates the value of system prompts: with the same user input, different system prompts can make the AI exhibit completely different personalities and behavioral patterns. This is also the key to building various specialized Agents (such as customer service assistants, code experts, and role-playing bots).

Summary: The Complete Path from Zero to Conversational Agent

Through this article, we've completed the most core part of an AI Agent—implementing conversation capabilities in code. Let's review the entire process:

1. Understanding the two core concepts: large language models and prompts;
2. Setting up a virtual environment and importing modules like OpenAI;
3. Referencing official documentation to call the DeepSeek API, with secure key management via .env;
4. Implementing continuous multi-turn interaction through a while loop;
5. Flexibly shaping the AI's persona and behavior with system prompts.

Although this is just a basic conversation program, it already has the embryonic form of an Agent. Building on this foundation, you can progressively add the following capabilities:

• Memory Management: Solving context window limitations through sliding windows, summary compression, or vector database retrieval, giving the Agent true "long-term memory" capability;
• Function Calling: Enabling the Agent to call external tools like search engines, execute code, or query databases. The technical implementation of Function Calling works as follows: developers declare available tools via JSON Schema descriptions in API requests (including tool names, parameter types, and function descriptions), the model determines when to call a tool during inference and outputs structured tool-calling instructions, external code executes the call and returns results to the model in role: tool message format for continued reasoning. The ReAct (Reasoning + Acting) framework formalizes this process as alternating cycles of "Thought → Action → Observation," enabling the Agent to dynamically adjust subsequent plans based on tool execution results. This is the theoretical foundation of mainstream Agent frameworks like LangChain Agent and AutoGPT, achieving a complete "perception-decision-action" closed loop;
• Multi-Agent Collaboration: Multiple specialized Agents collaborating on more complex task workflows. For example, a "Planning Agent" decomposes complex tasks into subtasks, multiple "Execution Agents" handle their specialized subtasks in parallel, and a "Synthesis Agent" integrates results—this pattern is called a Multi-Agent system and represents a frontier direction in current Agent research, with frameworks like AutoGen and CrewAI already providing mature multi-agent collaboration implementations. Notably, "inter-agent communication protocol" design is one of the core challenges in multi-agent systems: how agents pass task states, handle conflicting decisions, and prevent "hallucination propagation" (where one Agent's erroneous output is accepted as fact by other Agents) are active research directions in both academia and industry.

Deep Dive: Hallucination and Mitigation Strategies "Hallucination" in large language models refers to the model generating content that appears reasonable but is actually incorrect or fabricated—such as fake references, non-existent facts, or erroneous code logic. The root cause is that the model's training objective is "generating fluent, coherent text" rather than "outputting truthful information"—the model doesn't distinguish between known facts and plausible inferences, instead filling in content that statistically best fits the context. In multi-agent scenarios, the hallucination problem is amplified: hallucinated content from one Agent, if passed to the next Agent without verification, can create "error chain propagation." Common mitigation strategies include: RAG (Retrieval-Augmented Generation) introducing external knowledge bases as factual anchors, requiring the model to cite sources with secondary verification, deploying dedicated "verification Agents" in the Agent pipeline to cross-check critical information, and adopting confidence assessment mechanisms to flag low-certainty outputs.

For readers looking to get started with Agent development, getting this example running is the best first step.