LLM Streaming to the Browser: Buffering Pitfalls with FastAPI and SSE

When building web applications powered by large language models, a common frustration is this: streaming output works perfectly in the terminal, but in the browser, the entire response appears all at once. The model's streaming isn't the problem — the real culprit is some layer between your backend and the browser holding bytes back.

This article presents a complete solution for relaying streaming chat completions from a Python backend to a web page, and systematically breaks down all the places where buffering can silently creep in. The content draws from the hands-on experience of Aman Kumar, author of an OpenAI-compatible gateway.

Why Not Call the API Directly from the Browser

The most straightforward reason: your API key gets sent to every visitor along with your frontend code. Any key embedded in frontend JavaScript is effectively public — you're leaving your bill open to the world.

The correct pattern is to use your backend as a relay:

1. The browser sends the user's message to your backend;
2. The backend calls the LLM API with stream=True, using a key stored in an environment variable;
3. The backend forwards each chunk to the browser as it arrives.

This approach has additional benefits — your backend naturally becomes a centralized point for per-user rate limiting, usage logging, and model selection, all of which you'll need anyway.

Backend Implementation: FastAPI + OpenAI SDK

The official openai package can connect to any OpenAI-compatible endpoint simply by setting base_url. The recommended approach is to use the async client (AsyncOpenAI) so that a slow stream doesn't block other requests.

The core logic is an async generator gen() that iterates over the upstream stream, wraps each delta that contains content into an SSE event, yields it, and finally returns it via StreamingResponse with a text/event-stream media type.

There are a few details in the code that matter more than they might appear:

Wrap Each Chunk as JSON

Model output can contain newline characters, and bare newlines inside an SSE data: line break the frame structure. Encoding each text chunk as JSON completely sidesteps this issue. On the frontend, just JSON.parse when reading.

Skip Chunks with Empty Choices

Some providers send a final chunk containing only usage data with no choices. Indexing directly into [0] on such a chunk will throw an exception. Skip it with continue when choices is empty.

Detect Client Disconnection

If a user closes the tab, the upstream request will keep generating content unless you explicitly close it. On a pay-per-token API this is real money wasted; on a request-quota plan it burns your allowance. Check with await request.is_disconnected() and call await stream.close() promptly.

Send Errors as Events

Once a response has started with a 200 status code, you can no longer change it. So wrap exceptions into an explicit error event sent to the frontend, allowing it to display a useful message instead of hanging indefinitely.

---

SSE (Server-Sent Events) is a unidirectional HTTP-based push protocol. The server continuously sends an event stream to the client using the text/event-stream content type, with each event prefixed by data: and terminated by two newlines \n\n. Compared to WebSockets, SSE is lighter weight, naturally reuses HTTP connections, and supports automatic reconnection — making it ideal for LLM use cases where the server pushes continuously and the client only receives. An async generator (async def gen() with yield) is the idiomatic Python pattern for use with StreamingResponse: each yield of a byte string causes FastAPI to immediately write it to the TCP buffer and flush, rather than waiting for the entire function to complete — which is precisely what makes streaming work.

Frontend Implementation: Use fetch, Not EventSource

EventSource only supports GET requests and cannot carry a JSON request body, so for a chat endpoint that requires POSTing messages, the recommended approach is to read the stream using fetch.

The implementation uses res.body.getReader() to obtain a reader, then decodes chunks incrementally with TextDecoder. The most critical piece is the buffer variable buf:

A single network read may end in the middle of an event, or it may contain three complete events at once.

The solution is to split on the blank line between events (\n\n) and hold any incomplete tail in the buffer to be prepended on the next read. This "split and retain remainder" approach is what makes stream parsing reliable, preventing loss or corruption due to arbitrary network packet boundaries.

---

EventSource is the browser's built-in SSE client API — minimal to use, but it can only send GET requests and cannot set custom headers (such as Authorization). Chat applications typically need to pass conversation history in a POST request body, so you must use fetch to manually consume a ReadableStream. res.body.getReader() returns a ReadableStreamDefaultReader; each call to reader.read() asynchronously returns a {done, value} object where value is a Uint8Array. TextDecoder converts the bytes back to a UTF-8 string. Since the network layer delivers packets of arbitrary size, a single read() may contain only half an SSE message, so you must maintain a string buffer and split on \n\n boundaries rather than assuming each read corresponds to exactly one complete event.

Four Categories of Buffering Pitfalls

When streaming works fine locally but "just doesn't work" in production, it's almost always one of the following.

Nginx reverse proxy buffering. Nginx buffers upstream responses by default. Either send the X-Accel-Buffering: no response header (already included in the backend code), or set proxy_buffering off; for that location block. Also remember to increase proxy_read_timeout to prevent longer responses from being cut off at 60 seconds.

Compression middleware. Gzip-style middleware typically waits to accumulate a full chunk before sending, which completely breaks the streaming experience. Exclude text/event-stream from compression, or simply don't attach GZipMiddleware to streaming routes.

Serverless and some PaaS platforms. Some managed environments collect the full response before returning it. If the platform doesn't explicitly document support for streaming with your runtime, test it first with a minimal endpoint that emits a timestamp every second — before questioning your own code.

Cloudflare and other CDNs. Streaming usually passes through, but caching rules or response transformations can interfere. Cache-Control: no-cache helps, and make sure the route isn't matched by a "cache everything" rule.

There's one more easily overlooked trap: your own logging middleware. The author has seen middleware that reads the complete response body for logging purposes, effectively turning the stream back into a single chunk. Any logic that touches the response body must be stream-aware.

Using curl to Quickly Pinpoint Which Side Has the Problem

A single command can tell you whether the issue is on the frontend or the server side:

curl -N -X POST https://your-app.example/chat \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Count to 20 slowly"}]}'

-N disables curl's own buffering. If text streams in gradually here but not in the browser, the problem is in the frontend parsing. If it all arrives at once here too, investigate your server and proxy layers.

---

Understanding where buffering actually occurs helps you debug systematically. An HTTP response travels from the backend to the browser through multiple layers — the application layer (FastAPI/uvicorn), Python middleware, reverse proxy (Nginx), CDN/load balancer, the OS TCP stack, and finally the browser's rendering engine. Any layer can introduce latency by waiting to accumulate a full chunk before forwarding. The Content-Type: text/event-stream header combined with Cache-Control: no-cache is the standard signal to intermediate layers that "this is a real-time stream — do not cache or buffer it," but not all proxies respect this, so each layer needs to be verified individually. X-Accel-Buffering: no is a directive specifically for Nginx that can be set by the upstream application in its response headers without modifying Nginx config files — useful in managed hosting scenarios where you don't have direct server access.

Tips for Smooth Rendering

Appending plain text to a <div> on each token is fine. But re-rendering Markdown on every token will get progressively slower for long responses.

The recommended approach: display plain text during streaming, then do a single Markdown render when the done event arrives — or at most re-render once every ~100 milliseconds. There's also one non-negotiable safety rule: never insert model output as raw HTML. Always treat it as untrusted content to prevent injection attacks.

Rate Limiting Belongs in the Backend

Since every request passes through your backend, it's the ideal place for a simple rate limiter: maintain a per-user-per-minute counter in Redis or in memory, and check it before opening the upstream stream.

Whether you're paying per token or have a fixed monthly request budget, this protects you from having a single user (or a buggy frontend loop) drain your entire quota.

Pre-Launch Checklist

• API keys exist only on the server side.
• Every SSE event is JSON-encoded and terminated with a blank line.
• Chunks with empty choices are skipped.
• The upstream stream is closed when the client disconnects.
• Errors are emitted as events.
• Proxy buffering and compression are disabled for the streaming route.
• Before debugging the browser, test with curl -N first.

This setup works with any provider that speaks the Chat Completions format — just configure the base URL, API key, and model ID. It serves as both a ready-to-use engineering template and a systematic checklist for diagnosing streaming failures.