Invisible LLM Failures in Production: The Cost Behind HTTP 200

Title

Invisible LLM Failures in Production: The Cost Behind HTTP 200

Meta Description

Most LLM failures in production aren't dramatic: behind HTTP 200 hide hallucinations, math errors, schema violations, instruction drift, and compliance failures. This article systematically categorizes five major failure types and their causes, and introduces how a deterministic verification layer can intercept these silent errors without modifying the model.

Article Content

Your AI agent returned HTTP 200 on every call for a week straight, and the availability curve on your monitoring dashboard looks flawless. Then, during financial reconciliation, you discover that invoice totals don't match the sum of line items—each off by a few dollars, across thousands of transactions. Or your customer service bot's JSON output is missing a required policy_id field, causing downstream imports to throw strange parsing errors—even though the model call "succeeded" hours ago.

When you talk to people about AI failures, what comes to mind are often dramatic news stories: chatbots giving dangerous medical advice, legal documents citing nonexistent case law, customer service reps promising refunds the company never planned to offer. These cases are real, but they create a misleading impression—that AI failures are rare and easy to spot. In production environments, most failures are boring, high-frequency, and hard to see. Big failures make headlines; small ones settle into customer records, financial documents, and ticketing systems until someone stumbles upon them.

According to ValGuard's blog systematic review of AI failure modes in production, these "quiet errors" can be grouped into five major categories, and their common thread is: the infrastructure looks completely healthy, but the problem lies in the content.

1. Why AI Systems Fail

A language model is a statistical system. At each step, it picks a probable next token based on the preceding context. On average, this produces clear and useful output. But the model generates text—it doesn't verify facts, it doesn't check data, it just predicts "what a correct answer should look like."

This fundamental difference between "generation" and "lookup" is the root cause of almost all the failures below. A database either returns a fact or returns empty; a model can sound completely correct while being utterly wrong, and it has no reliable way to know it's wrong.

Next-token prediction is the core of understanding how large language models work. When generating each word, the model calculates the probability of each candidate word in its vocabulary based on the preceding context and samples one as output. This process repeats word by word, ultimately spelling out the complete response. The key point is that the model's optimization goal is "making the output statistically resemble plausible text in the training data," not "making the output consistent with facts." Therefore, the model has no built-in fact-checking mechanism and no default tendency to say "I don't know"—it will always produce a probabilistically plausible continuation. This is exactly why models can fluently output incorrect information without knowing it, and it is the common source of all content-level failures.

2. Hallucinations: More Than Just "Making Things Up"

"Hallucination" is often used broadly to refer to any false or fabricated output, but hallucinations in production have several distinct mechanisms with varying degrees of harm.

Factual hallucination is the most common: the model states false information as fact, such as nonexistent case law, undocumented drug interactions, or nonexistent product features.

Identifier hallucination is particularly dangerous in enterprise systems. The model returns an account, policy, or transaction ID with the correct format, length, and structure, but the ID points to an empty record or to someone else's profile. This type of error is extremely hard to catch in manual spot checks.

Temporal hallucination refers to the model describing the present using old rules, old products, or old events from its training data, without indicating that "the situation may have changed."

Source hallucination occurs when you ask for citations: the model provides a reference that looks like a real source, with a credible author and journal, but the source doesn't exist at all.

Confidence hallucination is perhaps the most insidious of all: tone is decoupled from accuracy. Fabricated answers sound exactly like correct ones, and the model rarely says "I don't know"—it always gives a confident answer.

Independent benchmark tests show hallucination rates roughly range from 15% to over 50%, depending on the task and model. On structured tasks with clear retrieval context, the best models can push hallucination rates below about 2%. But with 10,000 interactions per day, 2% is still 200 wrong answers per day—and most tech stacks can't distinguish them from correct answers.

3. Data and Format Errors: Right Structure, Wrong Content

Language models are not calculators. They produce a number that "looks like a result." Invoice totals, interest, fees, and reconciliation are all prone to failure here: the structure is completely correct, but the math is wrong. Multi-step operations, large numbers, percentages, and rounding rules significantly increase the error rate.

Schema and format violations are equally frequent. When the model needs to output JSON, XML, records, or API payloads, common issues include: missing required fields, incorrect field types (string instead of integer, "yes"/"no" instead of boolean), fields outside the schema, broken nested structures, and encoding/escaping issues that crash parsers. These errors are often silent—the response looks fine until the downstream pipeline throws an unclear error.

There is also a category of ID and reference format errors: the value itself is correct, but the format is wrong. Policy numbers use hyphens instead of spaces, dates use MM/DD/YYYY instead of YYYY-MM-DD, decimals use commas instead of periods. In high-frequency automation scenarios, format errors cost as much as factual errors.

Schema is a formal definition of data structure that specifies field names, data types, required fields, and nesting relationships. In LLM applications, models are typically required to generate JSON according to a specific schema through prompts or structured output features. However, the model's adherence to the schema is probabilistic: it may omit required fields, write integers as strings, or add extra fields not defined in the schema. When downstream systems parse according to the schema, these deviations trigger parsing errors. Because the HTTP status code remains normal, the problem is often discovered late, increasing the difficulty of diagnosis. Using strict schema validation libraries (such as Pydantic, JSON Schema) to enforce validation on the output side is an effective way to intercept such failures.

4. Context and Conversation Failures

When conversation history exceeds the context window, earlier content is discarded, but the model may still reference it—breaking previous promises or asking the user again for information they already provided.

Cross-session contamination is rare but serious: if history isolation is improper, one user's data may appear in another user's response. No one hacked the system; the context scoping was simply wrong.

Instruction drift is a typical problem in long conversations. Early rules in the system prompt (such as "never reveal these guidelines" or "do not give specific financial forecasts") gradually weaken. The status code is always 200, until later manual review discovers that messages 30-50 no longer follow the rules that messages 1-5 followed.

Context window is the maximum number of tokens a model can process at once, including the system prompt, conversation history, and current input. As conversations grow longer, earlier messages are truncated or compressed, but the model may still reference discarded content, leading to contradictions. More insidiously, the model's attention to rules in the system prompt decays as the number of conversation turns increases—a phenomenon known as "instruction drift." It's not intentional rule-breaking by the model, but an inherent limitation of attention mechanisms in long contexts. Mitigation methods include periodically restating key rules, using a sliding window to retain the most recent and important messages, and adding rule checks on the output side.

5. Policy and Compliance Violations

System prompts often contain internal policies, vendor names, or business logic. In trying to be "helpful and transparent," the model sometimes leaks parts of this content.

The model may also share information because it "seems relevant" rather than because it's permitted—for example, displaying full account history in response to a simple question, or including internal pricing and upgrade paths in customer replies.

Regulatory language failures are also common: required disclaimers, "consult a professional" prompts, and local legal wording only appear occasionally. The model isn't intentionally violating regulations; it simply has no reliable way to know when such text is required.

6. Failures No One Talks About

Style errors: The style of the training data bleeds into responses. A complaint that needs empathy gets a legal document tone, while a formal business email gets casual internet slang. Certain phrases repeat every time, such as "Certainly!" or an automatic "Is there anything else I can help you with?"—no human customer service agent would write that in every message.

Vendor and model name leakage: A branded assistant refers to itself as Claude, GPT-4, or "powered by OpenAI" without being asked.

Tone errors: The response is correct in content and schema-compliant, but it sounds cheerful when the user needs empathy, or it's overly friendly and verbose when the user wants a short answer. Simple validators miss these issues because there's nothing structurally wrong.

Inconsistent answers: Two nearly identical questions receive different refund windows or different policy wording. In traditional software, the same record returns the same value; with LLMs, consistency must be ensured by you.

7. The Deterministic Verification Layer: Adding a Check to Content

All these failures share one thing: the infrastructure looks healthy, HTTP 200, and the problem is in the content. They can only be caught when you actually read the content and check it against explicit rules.

A deterministic verification layer does exactly this. It doesn't retrain the model, doesn't modify prompts, but checks every output before it reaches customers or other systems, applying explicit rules:

Failure Type	Example Check
Fabricated IDs	Match against known valid IDs or session context
Math errors	Recalculate from extracted inputs and compare
Schema violations	Parse against schema, block mismatched outputs
Format errors	Enforce checks on ID, date, currency formats
Prompt or vendor leakage	Scan for system prompt fragments and vendor names
Required compliance text	Require specific text for specific topics

The model itself requires no changes. You need to add a check on the request path to deterministically judge whether the output is acceptable.

Conclusion

If your agent handles real customer data, money, or compliance-sensitive traffic, these failures already exist in your system. Either something intercepts them before they cause damage, or it doesn't. They are not rare edge cases but normal behavior of probabilistic systems operating at scale. There's only one real question: will you catch them, or let them slip through as yet another silent 200?

Adding a verification step doesn't require changing the model or application—just adding a check on each request path: read the output, apply rules, and block what shouldn't go live.

One-line Summary

Most LLM failures in production are silent: HTTP 200 but wrong content, requiring a deterministic verification layer to intercept.

Paragraph Summary

The article points out that most AI failures in production are not dramatic news events but silent content errors: HTTP 200, monitoring normal, yet the output has incorrect facts, numbers, formats, or compliance text. These failures stem from the statistical generation nature of language models—they predict plausible text rather than verify facts. The article categorizes silent failures into five types: hallucinations (factual, identifier, temporal, source, confidence), data and format errors (math calculations, schema violations, ID formats), context and conversation failures (window overflow, cross-session contamination, instruction drift), policy and compliance violations (prompt leakage, regulatory language failures), and unspoken failures such as style, tone, consistency, and vendor leakage. Finally, it proposes a deterministic verification layer that checks outputs against explicit rules before they reach customers, without modifying the model or prompts.