Routing LLMs by Prompt Length Backfired: A 12-Word Request That Broke a Production Agent

When building your own OpenAI-compatible chat gateway, tiered routing is almost a given: cheap models handle trivial calls, expensive models handle real reasoning. The question is — what metric should decide which path a request takes? One developer's production story reveals a seemingly reasonable heuristic that was hiding a serious trap.

A Routing Rule That Was Too Simple

The original approach was almost embarrassingly straightforward: route requests based on total prompt length. Anything under roughly 500 characters went to the economy (eco) tier; anything over 500 went to the premium tier. The rule passed every test case the author wrote, and looked airtight at launch.

Intuitively, it makes sense — longer prompts suggest more complex tasks, which naturally call for stronger models. But this conflates two fundamentally different dimensions: length measures cost, not capability requirements. How many tokens a request consumes and what capabilities the model needs to handle it correctly are two separate things.

How 12 Words Brought Down an Entire Agent Loop

The problem surfaced in production without warning. An agent started looping: the same request executed twenty times in a row, never converging.

When the author pulled up the call trace, the user's prompt was extremely short — "summarize this and call the report tool" — barely a dozen words. But that request came loaded with six tool definitions and a JSON mode flag.

The router only saw a short prompt, so it sent the request to the cheapest model available. That model was fine at writing prose, but unreliable when it came to tool calling: roughly half the time, it would respond in natural language instead of actually triggering any tool call. The agent treated these responses as failures, retried, failed again, and the loop was born.

Tool definitions are a core component of OpenAI's Function Calling protocol. Each tool definition contains a function name, description, and a JSON Schema for its parameters. The model must understand these structured definitions and, at the right moment, output a properly formatted call instruction — not plain text. This places high demands on a model's instruction-following and structured-output capabilities. The capability gap between models on tool calling is far wider than on ordinary text generation — some budget models perform acceptably on writing benchmarks but see a cliff-drop in accuracy in tool-calling scenarios. The JSON mode flag, which requires every model output to be valid JSON, similarly imposes a hard constraint on the model's format control. Both of these constraints represent "all-or-nothing" capability thresholds: a model either handles them reliably, or it fails in unpredictable ways — there's no middle ground.

A 70% Success Rate Is Effectively Zero Inside a Loop

The most valuable insight from this incident is about the compounding effect of probability within an agent loop.

Requests that carry tool definitions, structured output constraints, or image inputs have a hard capability floor — regardless of how short the text is, the model must possess the corresponding capability to respond correctly. A budget model might execute correctly 70% of the time, but inside an agent loop that requires repeated iteration, a 70% single-pass success rate compounds rapidly toward "never succeeds."

In other words, 70% sounds acceptable for a single call — but put it inside a loop that requires multiple consecutive correct steps, and that number collapses fast. This is the overlooked mathematical reality behind many agent systems that "randomly get stuck."

The math here is quantifiable: assume a single tool call success rate of 70%. An agent task requiring 4 consecutive correct tool calls has an overall success rate of 0.7 to the power of 4 — roughly 24%. If 6 consecutive steps are needed, the overall rate drops to about 12%. Real-world agent loops are often longer than this — when the model fails, the agent retries, and each retry consumes more of the success rate budget, quickly pushing the system into a negative feedback loop of "the more it retries, the more it fails." This phenomenon is known in reliability engineering as series system failure: unreliability at each link in the chain is amplified, not averaged out. This is exactly why agent systems place far stricter demands on underlying model capability than single-turn conversation scenarios — a success rate that's acceptable in a single call may be completely unusable in a multi-step automated pipeline.

The Fix: Capability First, Length Second

The actual fix took only an afternoon. The core change was adjusting the order of evaluation:

• Check capability flags first: Does the request declare any tools? Does it carry a JSON schema? Does it include images? If any of these are present, route directly to the lowest verified capable tier.
• Length only operates within a capability category: Once the capability tier is determined, length is used for further segmentation within that tier.
• Add a response validator: If a request declared tools but the returned response contains no tool calls, the router automatically retries at the next tier up — rather than dumping a pile of useless content back to the caller.

This "capability first, length second" approach essentially decomposes the routing decision from a single cost dimension into two layers: capability gating + cost optimization.

Results and Reusable Lessons

After the change went live, the loop rate for tool-using agents dropped from a "steady trickle" to near zero. More importantly, costs in the economy tier barely increased — because the vast majority of real traffic is chat prompts without tools, which still flow comfortably through the cheap path.

The author eventually packaged this routing logic into an OpenAI-compatible LLM Chat API, offering three tiers — auto, eco, and premium — with the auto tier encoding exactly this rule.

For developers building agent loops with tool calling, this lesson is worth keeping in mind: don't let a pure cost heuristic based solely on prompt length decide your routing. Cost can be optimized, but capability is a non-negotiable admission threshold. Make sure the model can do the job first — then figure out how to do it cheaply.