Don't Let json.loads Ruin Your LLM Pipeline: Diagnosing and Fixing Silent Failures

In production, an LLM isn't a chat toy — it's a part in a machine. But what it spits out isn't JSON; it's "text that looks like JSON." One engineer shares how he fixed silent JSON failures in his pipeline.

The Trap of json.loads

The code looks harmless:

data = json.loads(llm_response)
title = data["title"]

Concise, fragile. The model doesn't return JSON — it returns text. You ask it to fill in a form, and it replies with "Well, here's the thing…" — maybe useful, maybe completely off.

Five situations you actually encounter in production:

1. Markdown fences: The model wraps JSON in triple backticks, and json.loads blows up immediately. At least this failure is loud.
2. Politeness: Sure! Here is the JSON you asked for: {...} — blows up again. The model is eager to help; the pipeline isn't impressed.
3. Trailing commas, smart quotes: {"title": "Hello",} — the parser won't accept it.
4. Truncation: {"title": "Hello", "score": — incomplete JSON.
5. The worst — silent garbage: The model returns {"ok": true} instead of {"title": "...", "score": 42}. json.loads happily accepts it. The JSON is valid but the structure is wrong. Garbage flows downstream, discovered an hour later — or never.

Two Failure Modes, Not One

Type	Behavior	When Discovered
Loud failure	Throws, task terminates	Immediately
Silent failure	Wrong fields / wrong types	Too late

try/except handles the first. The second is what really hurts.

Things I Tried That Didn't Work

• "Tell the model to always return clean JSON." Sometimes works. But the model isn't a compiler.
• Regex matching between { and }. Breaks on nested objects and braces inside strings.
• Stacking a pile of if statements. Every new response structure = more ifs. Snowballing.

The author stopped begging the model and started checking it.

Schema validation means checking data against a pre-defined structural contract: which fields must exist, what type each field is, and whether values fall within allowed ranges. It's a separate concern from "can it be parsed" — json.loads only guarantees the text is valid JSON, not that the content matches your expectations. Common implementations include JSON Schema, Pydantic, Zod, dataclass validation, and so on. In LLM pipelines, schema is the critical gate that intercepts "silent garbage": {"ok": true} is perfectly legal syntax, but it's missing title and score, and only a schema can catch that. When designing schemas, explicitly declare required fields, types, and boundaries, and on validation failure, return field-specific error messages rather than a vague "validation failed."

The Fix

LLM text
   |
   +-- Strip Markdown fences and "Sure! Here's..."
   +-- Normalize quotes / trailing commas
   +-- Parse
   +-- Validate SCHEMA (types, fields, boundaries)
   |
   +-- If fail -> Repair loop
   |      Re-prompt with the exact error
   |      ("score: expected int, got str")
   |
   +-- If API is dying -> Circuit breaker
          Stop beating a dead horse

This mirrors the quality-control mindset on a factory floor: manufacture -> inspect -> repair or scrap. No wishful thinking. The author isn't from a software background; he spent years in real production environments. Parts go out according to the blueprint, get inspected, and rejects don't flow to the next process. LLMs are machines too — they just lie more confidently.

Circuit breaker comes from distributed systems' fault-tolerance patterns, first systematically described by Michael Nygard in Release It!. It monitors consecutive failures against a downstream dependency (here, the LLM API) and, once a threshold is reached, proactively "trips" to fail fast or take a degraded path instead of continuing to hammer requests through. After a cooldown period, it enters a "half-open" state, sending a small number of probe requests — success restores it, failure keeps it tripped. In an LLM repair loop, the circuit breaker matters because repair retries amplify request volume; if the model or API is already in a failure state, infinite retries only worsen the avalanche. Typical parameters: trip after 5 consecutive failures, 30-60 second cooldown, rate-limited probing in the half-open phase.

The Code (No Magic)

from llm_output_guard import Schema, Field, RepairLoop

schema = Schema(
    fields={
        "title": Field(str),
        "score": Field(int, min_value=0, max_value=100),
    }
)

def call_model(err=None):
    # err = exact validation error from the last attempt
    return generate(title=..., hint=err)

result = RepairLoop(schema=schema, max_attempts=3).run(call_model)

Repair isn't "roll the dice again" — it's feeding the error back to the model. In real runs, the second attempt fixes most type errors. You also need a circuit breaker: 5 failures -> cool down -> reset. Otherwise a clever repair loop becomes a hammer that utterly destroys a dying API.

Data from a Production System

• 49 Instructor injection points in one production pipeline, all going through the same JSON path
• 60+ schema contracts
• 12 behavioral tests specifically targeting "dirty" LLM output
• finish_reason=length counts as a failure, not a success

When you have dozens of calls, "it always parses eventually" is not a strategy.

finish_reason is a field returned in the response by most LLM APIs (such as OpenAI, Anthropic, etc.) indicating why generation stopped. Common values include stop (normal completion), length (truncated due to hitting the max_tokens limit), content_filter (interrupted by safety policy), and tool_calls (a tool call was triggered). Among these, length is the most critical: it means the returned text may have been cut off mid-sentence or mid-JSON, looking like it has content when it's actually incomplete. Treating length as success is the root cause of many pipeline incidents — downstream receives incomplete JSON, or retry logic repeatedly burns through quota. The engineering practice is to explicitly mark length as a failure, triggering a retry, shortening the input, or increasing max_tokens and re-requesting, rather than treating it as a normal completion and moving on.

Take This Checklist With You

If you take only one thing from this article — take this:

• Never call json.loads(llm_text) bare. Strip fences first.
• Always validate structure (types + required fields) after parsing.
• Treat finish_reason=length as a failure, not a success.
• On schema error: re-prompt with the exact error. Don't roll the dice.
• Put a circuit breaker around the API. 5 failures -> cool down.
• Log why validation failed. Future you will need it.
• Test with ugly output: fences, prose, trailing commas, wrong types.
• All call sites share one contract. Not 49 special cases.

Quick Pre-Release Check

Ask three questions:

1. If the model returns Markdown, what happens?
2. If a field has the wrong type, what happens?
3. If the API goes down mid-run, what happens?

If the answer is "crash" or "emit garbage" — it's not done yet.

What We Don't Do

• Don't send keys to random "JSON fix" services
• Don't treat json.loads(strip(text)) as sufficient
• Don't hide "almost passing" results

Key Takeaways

1. An LLM is not a JSON API. It's a text API. The JSON is your job.
2. Silent failures hurt more than crashes. Schema catches what try/except misses.
3. Repair with the exact error beats regenerating from scratch.
4. Circuit breakers protect you from your own stubbornness.
5. One contract serving all calls beats 49 snowflake special cases.

The author published these as small open-source libraries so others don't have to reinvent the wheel:

• structured-llm-output-guard — the safe JSON path
• guards-and-laws — quality gates + domain laws with hard vetoes
• checkpointed-task-pipeline — crash-resistant long tasks

If you're running LLMs in production, check your json.loads. If it has no schema and no repair mechanism — you might already be paying the price, just haven't seen the bill yet.

One-line Summary

LLMs return text, not JSON — schema validation and repair loops are essential defenses for production pipelines.

Paragraph Summary

Drawing on one engineer's hands-on experience, the article points out that in production, LLM output isn't JSON — it's "text that looks like JSON." Of the five common failure modes, Markdown fences, polite prose, trailing commas, and truncation cause loud parse crashes, while silent garbage with the wrong structure is the most dangerous — json.loads passes, but downstream receives wrong fields. The author's solution is a complete quality-control chain: fence-stripping and normalization, parsing, schema validation, a repair loop that re-prompts with the exact error, and a circuit breaker to prevent hammering a dying API. Production data includes 49 injection points, 60+ schema contracts, and 12 tests for "dirty" output, with finish_reason=length mandated as a failure. The core conclusion: an LLM is a part in a machine, and it needs inspection, repair, or scrapping like factory QC — not prayers for obedience.