LangGraph + FastAPI + SSE: A Practical Guide to Building Streaming AI Agents

Building an AI Agent isn't hard. What really tests your architectural skill is: how to expose it through an HTTP API, integrate real tools, and push the execution process to clients in real time. Based on a runnable open-source example, this article breaks down how LangGraph, FastAPI, and Server-Sent Events work together, and clarifies the key gaps between a demo and a production system.

Why Separate Agent Orchestration from API Transport?

LangGraph handles the Agent's execution flow: node scheduling, LLM inference, and tool calls. FastAPI handles exposing these capabilities as clear HTTP interfaces. These two responsibilities are entirely different, and mixing them makes the code hard to test and hard to evolve.

A useful mental model: the client doesn't need to understand how the Agent selects tools or manages internal state — it only needs to interact with a well-defined HTTP interface. The benefits of this separation go both ways: the Agent can be tested independently of the HTTP layer, and the API layer can remain unchanged when new clients are introduced later. Put simply, this means designing "orchestration" and "transport" as two systems that can evolve independently.

LangGraph is an Agent orchestration framework from the LangChain team. Its core abstraction is the "graph": nodes represent a computation step (such as calling an LLM or executing a tool), and edges represent the conditions for transitioning between steps. Compared to chained calls, a graph structure naturally supports loops, branches, and conditional jumps, making it well-suited for Agents that require multi-turn reasoning and tool calls. It has built-in state management — each node reads from and writes to a shared State, and the framework handles passing and merging state between nodes. This means the graph controls "who executes and how far execution has progressed," while the HTTP layer only needs to worry about how to trigger this graph and how to send results back — which is exactly the technical premise behind the separation of responsibilities emphasized in this article.

Giving the Agent a Real Tool: Weather Lookup

An Agent that can only generate text isn't very interesting. The example integrates a weather tool based on Open-Meteo, introducing a complete execution chain:

1. The user asks a weather-related question;
2. The model determines whether a tool call is needed;
3. The tool makes an external request to fetch weather data;
4. The result is returned to the Agent;
5. The Agent generates a response based on the result.

This chain looks simple, but it introduces problems that don't exist in pure LLM completion: network failures, tool errors, latency, and retries. Once your Agent needs to connect to a database or business system, these engineering constraints will surface in the same way. In other words, the weather tool is just a "minimum viable" sample of an external dependency.

Two Response Modes: Plain JSON and Streaming SSE

The FastAPI example demonstrates two response approaches. The plain JSON mode is suitable for simple integrations — the client sends a request and waits for the final result. But when Agent execution involves multiple steps, users may have to wait for seconds or even longer, which is where streaming responses come in.

Streaming mode is based on Server-Sent Events, which continuously pushes events through the HTTP response. A simplified SSE message looks like this:

event: message
data: {"content":"Processing your request..."}

It's worth noting that this only demonstrates the SSE format and is not the exact event contract in the repository. For AI applications, the value of streaming output lies in making long-running operations visible, thereby improving the interactive experience.

Server-Sent Events (SSE) is a one-way push protocol defined by the HTML5 specification. The client establishes a connection via a regular HTTP request, and the server continuously writes a text stream with the text/event-stream MIME type. Each message consists of several fields, the most commonly used being event (event type), data (payload), and id (event ID), with messages separated by blank lines. Compared to WebSocket, SSE is based on pure HTTP and requires no protocol upgrade, making it naturally suited for scenarios where "the server unidirectionally pushes progress or incremental results," such as AI streaming output. Its limitations are that it only supports server-to-client one-way communication, is subject to browser connection limits by default, and relies on client-side implementation for reconnection (continuation can be handled via Last-Event-ID).

Streaming Brings a Better Experience — and Engineering Challenges

SSE lets clients process events as they arrive, but once you move from "it works" to "reliable service," a series of questions emerge:

• How does the server handle a client disconnect?
• How are errors communicated to the client once the stream has already started?
• How does the client know the stream has ended?
• What happens if a tool fails mid-execution?

These questions can be avoided at the demo stage, but in production they directly determine system availability. The example repository is deliberately kept lean precisely so that you can see each component's responsibilities clearly before filling in these edge cases yourself.

Core Design Principle: The Agent Owns Execution; the API Only Handles Transport

One principle emphasized by the example: don't entangle Agent orchestration logic with HTTP transport logic.

The Agent should control its own execution flow and tool interactions, while the FastAPI layer only handles requests, responses, and streaming transport. The benefits go beyond testing — it reserves clear boundaries for introducing authentication, persistence, observability, and other features later. When you need to replace or upgrade a component, the scope of change is predictable.

What Hasn't This Example Solved Yet?

A working Agent API doesn't mean it's production-ready. Before using it for critical business, there are at least several categories of issues to confront.

Persistence and Checkpoints: Long-running or multi-step Agents may need to restore execution state after an interruption. LangGraph's checkpoint capability can come into play here.

Retries and Idempotency: When tools modify external systems, retries can have serious consequences. Take payment operations — you absolutely cannot double-charge due to a retry. The original article's reminder is key: checkpoints themselves don't guarantee idempotency; external side effects need a separate protection mechanism.

Observability: Production systems need to trace Agent execution, LLM calls, tool inputs and outputs, errors, and latency. LangSmith is an optional tracing solution in the LangGraph ecosystem that can cover execution, node, and tool-call levels.

Security and Operations Control: Authentication, rate limiting, secret management, structured logging, and deployment configuration all need to be implemented according to the specific environment.

These aren't reasons to say "don't build examples" — they're reasons to say "don't mistake a working foundation for a complete production system."

Idempotency means that executing the same operation once has the same effect on the system as executing it multiple times. In Agent scenarios, tool calls often carry external side effects — placing orders, charging payments, sending messages — and if they're retried due to a timeout or disconnect, they can cause duplicate operations. A common approach is to generate a unique idempotency key for each request, have the server record processed keys and directly return the previous result for duplicate requests, or rely on database unique constraints and deduplication tables as a fallback. It's especially important to note that LangGraph's checkpoint saves execution state — it guarantees "where to continue from," but it cannot prevent a tool from being called again. Therefore, external side effects must be guarded by an independent idempotency mechanism.

Try It Yourself: Run It, Then Swap the Tool

The open-source repository salada-dados/langgraph-fastapi-example contains the implementation code and setup instructions. It's recommended to start with the README, get the example running locally, and observe how the Agent, tool integration, FastAPI endpoints, and SSE streaming output fit together.

The most valuable next exercise is to replace the weather tool with an API relevant to your own business. During the replacement process, you'll naturally run into the operational issues that must eventually be addressed — which is exactly the design intent of the example.

Summary

This example establishes a working foundation with minimal code: LangGraph for orchestration, FastAPI for the interface, SSE for pushing, and Open-Meteo as a real external tool. Its value isn't in being feature-complete, but in clearly delineating each component's responsibilities. The gap between demo and production is real, but only by first building this foundation can you see where the gaps are.