Building a TypeScript Structured Output Validation Gate: Making LLM JSON Production-Ready

The LLM returned a JSON payload that looked fine. But when production needed a number, it got "high" instead.

This is the real challenge with structured output — it might pass once, but that's no guarantee it'll hold up every time. This article breaks down how to build a minimal "Schema Validation Gate" in pure TypeScript, turning the contract between probabilistic sampling and deterministic systems into something concrete.

Why Structured Output Needs a "Gate"

Structured output is having its moment, but vendor documentation openly acknowledges it isn't foolproof.

Anthropic's structured output docs implement JSON output via output_config.format and strict tool calls, but the page explicitly lists the failure modes we all know too well: parse errors, missing fields, type mismatches, and schema violations that trigger retries. OpenAI's structured output guide defines the capability as "schema-following" rather than just "valid JSON," and specifically calls out refusals and max-token truncation as two scenarios where responses may not match the schema.

The more telling signal comes from Anthropic's Claude Haiku 5.5 — a small model targeting high-throughput, cost-sensitive tasks like summarization, classification, and routing. The original tutorial makes a sharp point: when a small model is answering this frequently, a single bad structure is no longer an edge case — it's a rate.

Different products, same direction: vendors can constrain sampling, but your system still has to decide what happens when the structure breaks, gets truncated, or gets refused.

The original tutorial puts it well: a schema that passed once is a demo. A gate that can reject bad structure is a product.

---

"Constrained decoding" is the core mechanism vendors use to implement structured output, and it's worth explaining on its own. In standard sampling, the model picks the next token from a probability distribution over its vocabulary at each step. Constrained decoding adds a masking layer on top — at each position, only tokens that keep the output conformant with the target schema are permitted. For JSON output, if the current context requires a number, tokens like letters or quotes get their probability forced to zero. This mechanism statistically reduces format errors significantly, which is the underlying basis for why vendors like OpenAI and Anthropic can claim "schema-following" behavior. However, constrained decoding has two inherent blind spots: first, it can only constrain the legality of the token sequence, not its semantics — the model can still output wrong enum values or out-of-range numbers within a perfectly legal format; second, when output is truncated by max_tokens, the constraint layer stops with it, leaving a structurally incomplete response. This is exactly why even with vendor-side structured output enabled, an independent validation gate at the application layer is still necessary.

What This Gate Does

The example scenario is a routing decision with three fields:

• action: an enum, one of route, escalate, or ignore
• target: a string
• confidence: a number between 0 and 1

A simulated model returns eight responses: valid JSON, truncated JSON, wrong types, invalid enums, extra fields, missing fields, out-of-range confidence, and one "unfixable" stubborn sample.

The gate does exactly three things:

1. Parse
2. Validate against the schema
3. If invalid, feed the errors back for a single repair attempt, then accept or reject

One honest caveat worth flagging: the "model" here is a fixture table — all confidence values and pass rates are example inputs. In real usage, swap in your actual vendor call and keep the gate. The demo requires no API key, no real model, just TypeScript.

Core Design: Framework-Free Validation and Single Repair

Schema as Contract

The entire schema is just three fields: no extra properties allowed (additionalProperties: false), a closed enum, and a bounded number. The original tutorial emphasizes: if the schema itself is ambiguous, the gate will be too.

A Framework-Free Validator

The validation logic introduces neither Zod nor Ajv — it's hand-written checks for exactly what this schema needs: object type checking, required field checking, no extra properties, string and enum validation, numeric type and bounds checking.

The original tutorial makes a point that's easy to overlook: you can swap in a real JSON Schema library later. The interesting part isn't the library — it's what you do with the issues list once you have it. Validation without a decision is just logging.

Repair Once, Then Stop

The repair flow is deliberately kept short: parse → validate → if there are issues, make one request with the errors attached → validate again → stop.

In a real stack, mockComplete becomes your vendor call and firstIssues becomes the error payload you send back. Anthropic's own docs list schema violations as a reason for retrying — what this gate does is put that retry under your framework's control, rather than hoping it works inside a prompt.

One line from the original is worth remembering: one repair is a strategy. Infinite repair is an infinite loop. A repair loop with no rejection path is an endless apology.

---

Feeding error messages back to the model so it can self-correct is a pattern known in the LLM application space as "self-repair" or "feedback-driven regeneration." Its effectiveness comes from an empirical observation: most structural errors aren't caused by model incapability, but by random variance during first-pass generation. When a model is explicitly told "your output is missing field X, and the confidence value 1.3 is outside the 0–1 range," it can usually fix those specific, describable errors on the second try. But this mechanism has an important prerequisite: the error message must be precise and actionable. Vague feedback like "the output format is wrong, please retry" performs far worse than a structured list of specific issues. Furthermore, "repair only once" isn't conservatism — it's engineering rationality. Each repair consumes additional inference time and token cost, and empirical evidence consistently shows that the marginal benefit of multiple repair rounds drops off rapidly. Samples that still fail after one repair typically indicate ambiguity in the schema itself, or that the model genuinely cannot satisfy the constraints — continuing to retry at that point is just burning budget.

Using Three Metrics to Decide "Should This Ship?"

After running npx tsx gate.ts, the program outputs three key metrics:

• pass@1: the proportion of first responses that already conform to the schema
• pass@repair: the proportion of usable objects obtained after one constrained retry
• reject rate: the proportion that should be rejected outright

In the example run, only 1 out of 8 fixtures passes on the first try (pass@1 = 0.13), 6 become valid after repair (pass@repair = 0.88), and 1 stubborn sample remains broken and gets rejected. The gate's shipping threshold is pass@repair ≥ 0.75, so it outputs SHIP.

The original tutorial explicitly warns: the 0.75 threshold is just an example. You should derive thresholds from real traffic, not copy them from a blog post.

The spirit of the whole demo is condensed into one sentence: accept schema-conformant output, reject the rest, and count both.

---

The pass@k metric family originally comes from evaluation methodology in code generation — in benchmarks like OpenAI's HumanEval, pass@1 represents the probability of passing tests on a single generation, while pass@k is the probability of passing in at least one of k independent generations. This article adapts it to the structured output context with a slight semantic shift: pass@1 measures the proportion of first responses that conform to the schema, while pass@repair measures the cumulative proportion that are usable after one repair attempt. The adaptation is principled — both scenarios share the same essential structure: "after a limited number of probabilistic attempts, obtain an output that satisfies a deterministic constraint." Pairing these three metrics with an explicit shipping threshold transforms the go-live decision from "feels good enough" into a repeatable, auditable quantitative judgment — especially valuable for teams comparing model versions or vendors side by side.

Where It Will Fail

The original tutorial notably dedicates substantial space to limitations — which is especially valuable for engineering practice:

Vendor structured output is not your gate. Constrained decoding reduces the probability of bad structure, but it doesn't own your retry budget, manage your logs, or make your stop decisions. Keep the vendor features and keep the gate.

The micro-validator is not a full JSON Schema engine. This checker covers objects, required keys, enums, numeric bounds, and additionalProperties: false, but it doesn't support $ref, anyOf, arrays of objects, or regex patterns. Production schemas typically need a real library — but the pattern stays the same: issues come in, repair once, accept or reject.

Repair can "fabricate" a valid lie. Feeding errors back might cause the model to invent a target that never appeared in the original ticket. Schema-valid does not mean semantically true. For high-stakes fields, pair the gate with a whitelist or a human review step.

Eight fixtures is a smoke test. The 0.13 pass@1 here is an artifact of how the fixtures were written — it should be significantly higher on a constrained vendor path. A real test matrix should be built from production failures: truncated streams, enum drift, type coercion, and the extra keys your parser used to silently ignore.

The Bigger Picture

Structured output isn't a prompting trick — it's a contract between a probabilistic sampler and a deterministic system.

The full pipeline looks like this:

• Schema provides the shape
• Validator provides the evidence
• Repair provides one cheap second chance
• Rejection path provides the stop point
• Metrics provide the ship decision

When your structured output passes once, don't stop there — make it earn trust again on the next call. That's the dividing line between a fragile demo and a reliable product.