Andrew Ng on AI Agent Planning Pattern: How LLMs Autonomously Create Execution Plans

What Is the AI Agent Planning Design Pattern

In AI Agent development, a fundamental challenge has always existed: for complex tasks, must developers hardcode every step of execution in advance? Andrew Ng addresses this directly in his prompt engineering course on Agent design patterns — through the "Planning Design Pattern," large language models can autonomously determine the steps needed to complete a task, without requiring manually predefined workflows.

This article systematically covers how the planning pattern works, its typical use cases, and the real-world challenges it currently faces.

AI Agents are one of the core paradigms in modern LLM applications. Unlike single-turn Q&A, Agent systems give LLMs "action capabilities" — using external tools (APIs, databases, code executors) to complete complex tasks requiring multi-step reasoning. The Tool Calling/Function Calling mechanism was formally introduced by OpenAI in GPT-4, allowing models to "declare" the need to call a function during inference, with the result returned to the model to continue reasoning. This human-machine collaborative loop forms the foundation of modern Agent architecture.

The core idea of the planning pattern is: give the LLM a set of callable tools, let it autonomously generate a step-by-step execution plan, and then execute it incrementally to complete complex tasks.

Unlike traditional "hardcoded workflows," the planning pattern doesn't require developers to predetermine the order of tool calls. In traditional software engineering, workflows are predefined directed graphs or state machines — each node's predecessors and successors are fixed before deployment. This approach offers strong predictability and debuggability but is extremely rigid. The planning pattern transfers orchestration authority from the developer to the model itself, leveraging the LLM's "commonsense reasoning" to dynamically generate an execution path suited to the current task. This brings tremendous flexibility — the same Agent can handle many different types of requests without writing separate logic for each scenario.

Andrew Ng illustrates this with a sunglasses retail example. Suppose you run a sunglasses shop with inventory in a database, and a customer asks: "Do you have round sunglasses in stock for under $100?" This query is complex: it requires checking product descriptions to identify round sunglasses, verifying stock availability, and filtering for items priced under $100.

To handle such requests, the developer equips the LLM with tools such as: get product description, check inventory, process returns, query price, view transaction history, and handle sales. A prompt is then written telling the model which tools are available along with their descriptions, and asking it to "return a step-by-step plan to fulfill the user's request."

How the Planning Pattern Executes Step by Step

The key to understanding the planning pattern lies in its "step-by-step execution" mechanism. Academically, this is closely related to the ReAct (Reasoning + Acting) framework, proposed by Google researchers in 2022. ReAct has the LLM alternate between "Reason" and "Act," forming a chain-of-thought-driven tool-calling loop. The planning pattern extends ReAct further — the model not only reasons at each step but generates a complete execution plan (Plan) at the start of the task before proceeding. This aligns with the "Plan-and-Execute" architecture (e.g., LangChain's Plan-and-Execute Agent), with the distinction lying in plan granularity and whether mid-course dynamic revision is allowed.

Using the round sunglasses query as an example, the LLM might output a three-step plan:

Plan Generation and Execution

1. Step 1: Use the "get product description" tool to check all product descriptions and identify round sunglasses.
2. Step 2: For the filtered round sunglasses, use the "check inventory" tool to verify availability.
3. Step 3: Use the "get price" tool to confirm which in-stock items are under $100.

Once the LLM outputs this plan, the system passes the Step 1 instruction text along with tool descriptions, the user request, and background context back to the LLM to execute Step 1. The model calls the "get product description" tool to filter round sunglasses.

Then, Step 1's output is combined with Step 2's instruction and passed back to the LLM for inventory checking; Step 2's output is combined with Step 3's instruction to query prices; finally, all results are compiled and fed back to the model to generate the final answer for the user.

Andrew Ng specifically notes that plans generated by the LLM in practice are typically much more detailed than single-line instructions, containing multiple steps and contextual information such as task descriptions and available tools. The most exciting aspect of this approach: we don't need to decide the order of tool calls in advance.

If the customer makes a different request — say, "I'd like to return the gold-frame glasses I bought before, but not the metal-frame ones" — the LLM can autonomously generate a different plan: first query purchase history, locate gold-frame glasses via product description, then call the return tool. This is the power of the planning pattern: a single Agent can cover a wide range of tasks that require calling multiple tools in varying orders.

Email Assistant: Another Typical Use Case for the Planning Pattern

The course also presents an email assistant example to further demonstrate the pattern's versatility.

Suppose you tell the assistant: "Please reply to the dinner invitation email from Bob in New York, tell him I'll attend, and archive the email." The assistant is equipped with tools for searching emails, moving emails, deleting emails, and sending emails.

The LLM might plan the following steps: use the "search email" tool to find Bob's dinner and New York email, generate and send a confirmation reply, then move the email to the archive folder. The system then executes step by step: search for the email, send the reply, and complete archiving after confirming the send was successful.

This example makes clear that the planning pattern is not limited to e-commerce queries — any task requiring multi-tool collaboration can adopt this paradigm.

Real-World Applications and Limitations of the Planning Pattern

The planning design pattern has already found successful application in highly autonomous coding Agent systems. Andrew Ng notes that when you ask such systems to build a moderately complex application, they first create a plan — build this component, then that component — essentially forming a checklist, which they then work through item by item.

Coding tasks are especially well-suited for the planning pattern because code execution provides clear "right/wrong" feedback signals. Systems like Devin (Cognition AI), GitHub Copilot Workspace, and OpenDevin heavily adopt this approach — given a request like "develop a web app with user authentication," the system generates a modular development plan (design database schema, implement backend API, write frontend components, configure deployment scripts) and executes it incrementally. The system can run tests to verify each step, forming a "plan—execute—verify—correct" closed loop that significantly reduces the risk of uncertainty.

Current Challenges Facing the Planning Pattern

However, outside of coding scenarios, the planning pattern remains largely experimental and has not yet been widely adopted. Andrew Ng candidly identifies the core challenge:

The planning pattern makes systems harder to control. As a developer, you don't know at runtime what plan the model will generate. The fundamental reason it's "hard to control" is that LLM output space is too large — in theory, the model can generate any sequence of tool calls, including dangerous operations (e.g., bulk-deleting data, sending unauthorized emails). This uncertainty is a genuine concern for production environments that require high reliability.

The industry currently employs several main mitigation strategies: "Human-in-the-loop" — pausing before high-risk steps to request user authorization; "sandboxed execution environments" — restricting the Agent's operational permissions to an isolated environment; and "Plan Critic" layers — using a second model or rule engine to perform a safety review of the generated plan before execution. These strategies seek a balance between flexibility and controllability, and are central topics in current Agent safety research.

For these reasons, overall adoption of the planning pattern — outside the coding Agent domain where it excels — is still in a growth phase. Andrew Ng remains optimistic, believing the technology will continue to improve and appear in an increasing range of application scenarios.

Summary

The planning design pattern represents an important direction in AI Agent development: shifting from human-orchestrated workflows to model-driven autonomous decision-making. Its value lies in flexibility — one set of tools and one planning prompt can handle a wide variety of complex tasks without coding separate logic for each scenario.

For developers, understanding the core mechanism of "generate a plan, execute step by step" is foundational to building autonomous Agents. How to strike the right balance between flexibility and controllability is the key challenge in current real-world deployment. As LLM capabilities continue to advance, the planning pattern is poised to demonstrate broader value beyond the coding domain.