The Nine-Step Anatomy of an LLM Request: The Full Chain from Click to Response

Most developers, when they first integrate an LLM, care about only one thing: call the API and get the result. But after systematically mapping out the LLM application request chain, one frontend engineer discovered that what really determines success or failure is often not the model itself—of the nine steps, only two belong to the model, and the remaining seven are all engineering contracts.

Why break down an LLM request

The author has 8+ years of React/TypeScript frontend experience and is publicly learning applied AI, with the goal of becoming an AI engineer or even an AI Forward Deployed Engineer (FDE). Before diving into tokens, RAG, and agents, he first drew a map: when a user clicks the "Summarize" button, what actually happens inside the system?

This shift in perspective matters. To a frontend engineer, an LLM feature looks like nothing more than "send text to the backend and wait for the result to come back," but hidden in between are a series of engineering decisions around validation, budgeting, streaming, and error handling. Breaking these apart helps teams avoid pitfalls when integrating AI.

Before the call: steps 1-4

The client sends the request; the key must stay on the server

From the browser's perspective, an LLM feature is just an ordinary HTTP call: POST /summarize, carrying the user's text, plus a session id for chat scenarios. OpenAI's official documentation explicitly requires that API keys never be exposed in client-side code, so the key lives on the server, and only the server communicates with the model provider.

Input validation is the cheapest place to stop losses

FastAPI uses Pydantic models to validate the request body. Invalid data never even reaches the business logic function: FastAPI raises RequestValidationError, and the default handler returns HTTP 422. Once validation passes, Pydantic guarantees the field types match the declaration. This is the lowest-cost place to block bad input—no tokens have been consumed yet.

Prompt assembly: statelessness is the default

The model can only see what you send it. Anthropic's Messages API is stateless, so the full history must be resent every time; OpenAI likewise treats each request as independent and stateless, and although it offers optional conversation state management, it explicitly notes that earlier input tokens in the chain are still billed on every turn.

Token counting and budget checks

The model reads tokens, not characters, and the tokenizer is bound to the model, so counts differ across models. The system prompt, tool definitions, history, and output all share the same context window. If the input alone exceeds the limit, the request fails outright (Anthropic returns 400 with "prompt is too long"). Estimating token counts before sending and setting a maximum output length are the basics of cost control.

Inside the model: steps 5-6

Prefill: reading in all input in parallel

Inference has two phases. In the prefill phase, the model processes all input tokens in parallel; it is compute-intensive and is the main model-side factor determining time to first token (TTFT). Network transmission and provider queueing time add on top of that pause.

The KV cache (Key-Value Cache) is the key mechanism for understanding decode-phase performance. In the Transformer's self-attention computation, each token generates corresponding Key and Value vectors, and these vectors need to be reused repeatedly when generating subsequent new tokens. If the K and V of all historical tokens were recomputed at every step, the computation would grow quadratically with sequence length. The KV cache approach stores the already-computed K and V matrices in GPU memory; when generating a new token, it computes only the current token's K and V and appends them to the cache, making the computation per decoding step approximately constant. The cost is that memory usage grows linearly with context length, which is one reason long-context requests are more expensive. The prefill phase is precisely when this cache gets filled, which is why it is compute-intensive; the decode phase, thanks to the cache, becomes relatively lightweight but is limited by memory bandwidth.

Decode: token-by-token autoregressive generation

The model then generates autoregressively: score every token in the vocabulary, pick one, append it, repeat. The KV cache retains prior computations so it doesn't start from scratch at each step, and the cadence of each step is the time per output token. Generation stops when it hits an end token, a stop sequence, or the token limit.

As for "how to choose," that falls under sampling settings: greedy, or sampling with temperature, top-k, and top-p. Don't assume these parameters are available on all models—Anthropic's documentation states that Claude 4.7 and later return 400 if these parameters are set to non-default values.

SSE (Server-Sent Events) is an HTTP-based server push technology that allows a server to continuously send a stream of text events to a client over a single connection. Unlike WebSocket, SSE is unidirectional (server to client), uses ordinary HTTP, and is naturally compatible with existing infrastructure and proxies. In the LLM context, providers wrap each generated token as a data: event and push it incrementally, and the client reads it via EventSource or streaming fetch. The two pitfalls mentioned in the article stem precisely from SSE's protocol characteristics: once response headers are sent with 200, any subsequent error can only be conveyed as an in-stream event, and the HTTP status code can no longer reflect it; at the same time, streaming output means users and moderation systems see partial content before generation finishes, creating timing challenges for content safety review.

After the call: steps 7-9

Streaming responses: errors can hide behind a 200

Both OpenAI and Anthropic stream tokens via SSE during decoding, so steps 6 and 7 overlap. The documentation contains two easily overlooked pitfalls: errors can arrive mid-stream, at which point the status code is already 200; and partial output is harder to moderate than a complete response.

Output must be validated too

Structured outputs can constrain the model to follow a JSON Schema you provide (OpenAI supports a subset), but its guide also acknowledges that errors are still possible and lists two cases that must be handled: refusal responses and responses truncated by the token limit. If you use streaming, validate only after the stream ends. FastAPI's response_model is the second check: a type mismatch becomes a server error rather than returning bad data. But remember: valid format does not equal a correct answer—correctness evaluation is a topic for later.

Retry case by case, and log everything

The same status code can mean different things: 429 is usually rate limiting, but Anthropic also returns 429 when you hit your monthly spend limit, so read the error body before retrying. Never retry quota or billing errors. For transient errors, respect Retry-After if present; otherwise use exponential backoff with jitter, and cap both the number of retries and total duration. The official SDKs already retry some errors (Anthropic defaults to two), so factor that into your own retry ceiling instead of nesting loops layer upon layer.

Finally, log input/output token usage, latency, and the provider's request id. These records support debugging and cost tracking, and ultimately the evaluation system too.

Exponential backoff with jitter is the standard strategy for handling transient failures in distributed systems. The basic idea is that the wait time before each retry grows exponentially (e.g., 1s, 2s, 4s, 8s), avoiding a sustained high-frequency barrage while the server is overloaded. But pure exponential backoff has a hidden risk: multiple clients may retry at the same moment, creating a "thundering herd" effect that overwhelms the service again. Jitter disperses retry moments by adding randomness to the backoff time; common forms are full jitter (random between 0 and the computed value) and equal jitter (random between half the computed value and the full value). When used together with the Retry-After response header, you should prioritize the server's suggested wait and fall back to your own backoff strategy only when it is missing, and always set a maximum retry count and total timeout to prevent infinite retries from dragging down the request chain.

The core principle: treat the model as an unreliable, metered remote dependency

As a frontend engineer, the author finds this whole flow unexpectedly familiar: it's a request lifecycle, just with an unusual backend swapped in. Of the nine steps, only two belong to the model; the other seven are contracts, budgets, streams, and failure handling—exactly the parts engineers control.

His working rule follows from this: treat the model as an unreliable, metered remote dependency, and put validation on both sides of it.

This judgment has practical significance for teams currently integrating LLMs. Many projects concentrate effort on prompt tuning, only to stumble on input validation, token budgets, streaming error handling, and retry strategy. Model capabilities will keep evolving, but the engineering constraints of the request chain will not disappear.

The author's learning roadmap places this piece in the fundamentals module: Python typing and async, FastAPI and Pydantic, and LLM mechanics (tokens, context, inference). The next post will focus on "Python for AI backends: a refresher course for TypeScript engineers." He also poses a question to readers: for the first LLM feature you built, which of these nine steps broke first?

One-line Summary

Of the nine steps in an LLM request, only two belong to the model; seven are engineering contracts, and validation and budgeting are what really determine success.

Paragraph Summary

A senior frontend engineer systematically breaks down the complete request chain of an LLM application into nine steps and points out that only prefill and decode belong to the model's internals; the other seven—client forwarding, input validation, prompt assembly, token budgeting, streaming, output validation, and retry logging—are all engineering contracts. The article emphasizes that API keys must stay on the server, Pydantic validation is the cheapest place to stop losses, and model APIs are stateless by default, causing history tokens to be billed repeatedly. In streaming output, errors can hide behind a 200 status code, and a 429 may mean rate limiting or exhausted quota. The core principle is: treat the model as an unreliable, metered remote dependency, and put validation on both sides of it.