Six Layers of Defense: Engineering Reliable Structured Output from LLMs

If you've built any Agent system, you've probably run into this: you ask an LLM to return some JSON, and it starts with "Sure, here's the result you asked for" — or drops a stray comma at step 49, crashing the entire pipeline. It's like ordering takeout with a note saying "no cilantro" and getting a sticky note back that reads "but cilantro is delicious."

A one-off failure is easy enough to retry, but in an Agent system, every step's output feeds the next step's input. One malformed response means a parsing exception, and the whole task falls apart. That's the fragility of the "input → output → input" loop. This article presents a systematic six-layer defense strategy — from prompt-level guidance to hard constraints, from single-step fixes to full-pipeline guarantees — to keep your "rogue model" firmly in line.

Root Cause: Models Are Born to "Improvise"

At its core, an LLM is a token generator. Its training objective is to produce "text that looks natural," not "precise data structures." So most of the time it returns valid JSON — but occasionally it adds an extra comma, drops a quote, or writes a number as a string. This isn't intentional misbehavior; it's baked into the training objective.

The key insight is: structured output can't be coaxed out with prompts alone — you need an engineered system of validation and error correction. If your answer in an interview is just "add a JSON Schema," the follow-up will immediately be: "What if it still outputs garbage? How do you guarantee compliance across 50 Agent steps?" Stumbling there reveals a shallow understanding of production engineering.

Layer 1: Constrained Decoding — Block Format Errors at the Physical Level

Constrained decoding is the lowest-level, most hard-core defense. The idea is straightforward: at each decoding step, the model is restricted to only choose from a set of tokens that are valid according to a predefined JSON Schema.

For example, the first character of a JSON object must be {. So the decoder physically blocks every other token — [, ", Chinese characters, everything — leaving the model with no choice but to emit {. Think of it like a multiple-choice exam: every blank has fixed options, eliminating any chance of error.

Under the hood, constrained decoding relies on finite state machines (FSMs) or context-free grammars (CFGs). At each time step, the decoder maintains a "current valid token set" determined jointly by the generation state and the Schema. Open-source frameworks like Outlines, llama.cpp's grammar sampling, and vLLM's guided decoding all provide this capability. It's worth noting that the computational cost of constrained decoding isn't uniform — for deeply nested JSON or complex Schemas, computing the valid token set can itself become a bottleneck. In practice, you'll need to balance Schema complexity against performance.

This is fundamentally different from a prompt-level "please return JSON" — one is a request, the other is enforcement. Major APIs including OpenAI and Anthropic now offer strict mode: when enabled, tool call parameters precisely match the Schema, eliminating type mismatches and missing fields at the root.

Side Effect: Quality Degradation

As powerful as constrained decoding is, it comes with a real trade-off: quality degradation. When hard constraints block all the high-probability tokens the model would naturally prefer, it's forced to pick from low-probability alternatives — resulting in output that's "syntactically perfect but semantically nonsensical." For instance, the rating field might be perfectly formatted but hold the value 100, or a price field might say "very expensive" as a string.

Research has noted that base models generally benefit from constraints (cleaner, more direct output), but instruction-tuned models can suffer meaningful quality drops on open-ended generation tasks. The core takeaway: constrained decoding guarantees format correctness, not content correctness. It's the first line of defense, not the last.

Layer 2: Validation and Retry Loop — Enforce Semantic Correctness

Constrained decoding can guarantee that rating is an integer — but if your business rule says it must be between 1 and 5 and the model outputs 10, constrained decoding won't catch that. That's where the second layer comes in: validation and retry loops.

Use Pydantic or JSON Schema to define types, value ranges, and cross-field logic. After the model produces output, run it through a validator to keep LLM uncertainty within business-safe boundaries. Syntax gets one layer, semantics gets another — together they're complete.

Pydantic and JSON Schema have become the de facto standard in LLM engineering. Pydantic auto-generates JSON Schema from Python type annotations and enforces data types, range constraints, regex matching, and more at runtime. Tools like LangChain's with_structured_output and the Instructor library both use Pydantic models as their core abstraction, merging "define the expected output structure" and "call the model" into a single operation. JSON Schema itself is an IETF standard, supporting minimum/maximum, enum, pattern, $ref, and other rich constraint expressions that cover the vast majority of business semantic validation needs.

When validation fails, send the error message (e.g., "rating value is 10, out of the valid range 1–5") back to the model along with the original output, and ask it to regenerate. This creates a "validate → feedback → generate" loop until the output passes. In production, always set a maximum retry count (e.g., 3) to prevent infinite loops.

Validation-and-retry isn't a replacement for constrained decoding — it's a complement: constrained decoding handles syntax, validation-and-retry handles semantics. It also decouples business rules from model calls — updating a rule means changing the validator config, not the model code — and validation failure logs provide valuable observability.

Layer 3: Fake Tool Calls — Low-Cost Stability Boost

This is a clever trick that can quickly improve output reliability. Modern models are fine-tuned to be very good at generating function call arguments to a specified Schema. We can exploit this: define a dummy tool without actually calling anything, write the expected output format as its input parameters, then force the model to "call" this tool. It will dutifully output structured JSON matching the Schema.

This approach has a low technical barrier — virtually all major models support tool calling, and Claude's Agent SDK and OpenAI's strict mode both support it out of the box. Some teams have used this technique to push JSON output stability from 70% to over 95%.

Layer 4: Logit Masking — Precise Control Over Tool Selection

Starting here, we enter problems unique to Agent systems: choosing the wrong tool. At each step, an Agent selects from a pool of tools — and the more tools there are, the higher the chance of picking the wrong one (e.g., calling search when it should use a calculator).

You might think "just add or remove tools dynamically" — but this has a nasty gotcha: tool definitions sit at the very beginning of the context. Modifying them invalidates all downstream KV Cache, forcing the model to recompute massive amounts of attention. Latency and cost spike dramatically, making this approach nearly unworkable in practice.

Understanding KV Cache and tool definitions is important here: KV Cache is a core inference optimization in Transformer models. For already-computed token sequences, the Key and Value matrices are cached and reused during new token generation, avoiding redundant computation. Since tool definitions appear at the front of the system prompt, any change to them invalidates all KV Cache from that point forward. In long-context scenarios — say, an Agent 50 steps in with tens of thousands of tokens in context — modifying tool definitions triggers massive recomputation, pushing latency from milliseconds to seconds and multiplying API costs. This is precisely why Manus chose the "never change tool definitions + Logit Masking" strategy.

Manus's approach is Logit Masking: all tool definitions are fixed from the start and never modified. At decode time, they mask logit values to remove currently irrelevant tools from the probability distribution. They also designed consistent tool naming conventions — browser tools prefixed with browse_, shell tools prefixed with shell_ — so prefix-based masking is both efficient and fully KV-Cache-safe.

Layer 5: Schema Contracts — Structured Communication Between Agents

When a task is distributed across multiple Agents, communication between them can't rely on vague natural language — it requires explicit Schema contracts. Think of it like two teams integrating via a formal API spec rather than word of mouth. Who produces data, what it looks like, and who consumes it — every step is defined clearly. Even if an Agent's internal logic changes, the system stays stable as long as the contract holds.

Manus uses a MapReduce-inspired pattern: a Planner Agent first defines the output Schema, multiple sub-Agents fill in data in parallel, and results are merged at the end. MapReduce was originally proposed by Google in 2004 as a classic paradigm for large-scale parallel data processing — the Map phase distributes tasks to Workers for parallel processing, and the Reduce phase merges results. In multi-Agent systems, this idea extends naturally: an Orchestrator Agent handles task decomposition and Schema definition, multiple Sub-Agents execute homogeneous subtasks in parallel, and results are merged according to the predefined Schema. For example, extracting information from 100 resumes can be handled by 10 Sub-Agents running simultaneously and then aggregated — highly efficient. The key here is that subtasks are fully decoupled: a single Sub-Agent failure won't cascade to others. However, the Reduce phase's result-merging logic requires careful design, especially when subtask results have dependencies or conflicts.

OpenCode's context compression mechanism follows a similar philosophy: rather than arbitrarily truncating context, it outputs a fixed-format summary containing goals, key instructions, completed/pending work, and relevant file lists — ensuring critical information is complete and consistently formatted after every compression.

Layer 6: Anti-Pattern Locking — Hidden Traps in Long-Running Agents

This is the most commonly overlooked layer. After dozens of steps, the context fills up with large amounts of similar actions and observations. The model starts mechanically repeating itself, falling into a habitual loop — like an employee who's done the same task so many times they go on autopilot and can't adapt when something new comes up. Manus encountered exactly this when processing resumes in bulk: the model kept repeating "open document, extract information" for every resume, without adjusting strategy for unusual formats.

Pattern locking in autoregressive models has a deep mechanism: in autoregressive generation, each new token's probability distribution is conditioned on all previously generated tokens. When the context is saturated with similar action-observation pairs, the model's conditional probability gets strongly pulled by historical patterns, creating a "pattern attractor" — even when the current situation calls for a different strategy, the model tends to repeat established patterns. This closely parallels what psychologists call "Functional Fixedness." From an information-theoretic perspective, repetitive patterns in long contexts reduce the entropy of new token predictions, making it harder for the model to "escape" the current pattern.

This failure mode is subtle because the model doesn't throw errors — it's "too obedient" to past patterns. There are two approaches:

• Micro-variation: Keep the Schema unchanged, but vary the serialization slightly (randomize field order, swap out description wording) to break the autoregressive momentum.
• Context reset: Periodically compress context or perform a Context Reset at key checkpoints — for example, summarize every 10 steps and retain only the most critical information. Context compression and periodic resets fundamentally restore the model's exploratory capacity by reducing historical bias.

Two Easily Overlooked Engineering Details

First, field descriptions in a Schema are instructions for the model, not documentation for humans. There's a world of difference between "rating: integer" and "Score from 1 (very poor) to 5 (excellent) based solely on content quality." A good description is like onboarding training for the model — the clearer it is, the better the output.

Second, separate reasoning from output. Don't apply hard constraints from the start. Let the model think freely first, then apply structural constraints only when producing the final output. Claude already implements this — unconstrained thinking, structured final output. An even better practice: first round lets the model reason freely; second round takes that reasoning as input and requests structured output. It requires an extra call, but the quality improvement is significant.

Beware of Over-Constraining: Don't Weld Your Architecture Shut

Manus founder Peak offered an important caution: all the structural constraints we add today genuinely improve stability in the short term — but models are advancing rapidly, and constraints that are necessary today may become bottlenecks tomorrow.

The test is simple: run the same Agent evaluation suite on models of different capability levels. If swapping in a stronger model doesn't produce a meaningful improvement, your control mechanisms may be holding the model back. GPT-3.5 needed tight constraints; GPT-4 might produce great output with far fewer restrictions. The recommendation: revisit and recalibrate every six months, or with every model upgrade, to find the new balance between safety and performance.

Summary

These six layers — constrained decoding, validation-and-retry, fake tool calls, Logit Masking, Schema contracts, and anti-pattern locking — form a complete engineering framework: from hard constraints to soft validation, from single-module fixes to multi-module coordination, all the way to long-term runtime stability. This isn't just a talking point for interviews — it's a reusable reference for designing any production-grade system that depends on LLMs.