How Constrained Decoding Makes LLM Outputs Strictly Follow a Schema

When large models call tools or return structured data, the stability of JSON formatting has always been a pain point for engineering deployment. Constrained Decoding is seen as the key solution to this problem, but a widely overlooked fact is: it can only guarantee the "shape" of the output, not the correctness of the output's "content."

What Constrained Decoding Actually Does

The core mechanism of constrained decoding is to apply a mask to the sampling process during the token generation stage. At each step, the model outputs a logits distribution, and the decoder calculates which tokens are legal in the current state based on a predefined grammar (usually a JSON Schema or some form of formal grammar), then directly sets the probabilities of illegal tokens to zero and samples only from the legal candidate set.

This way, no matter what the model's internal probability distribution looks like, the final output token sequence is guaranteed to be legal at the syntactic level. Correct field name spelling, complete bracket closure, accurate comma placement—these problems that require repeated retries or even post-processing fixes under bare sampling are eliminated from the very first step of generation under constrained decoding.

From an engineering perspective, this does solve a large class of problems. In scenarios such as function calling, structured extraction, and API parameter generation, the format compliance rate can be raised from "mostly correct" to nearly 100%.

Logits and Sampling: Each time a large model generates a token, it outputs a set of real-valued scores for the vocabulary (usually tens of thousands to over a hundred thousand entries), called logits. These scores are normalized via softmax into a probability distribution, and the decoder then samples the next token based on that distribution. The so-called "mask" means setting the logits corresponding to illegal tokens to negative infinity (equivalent to setting their probability to zero), so that sampling only proceeds within the legal candidate set. Understanding this point is crucial: constrained decoding does not change the model's "thoughts" about the rest of the vocabulary; it only performs filtering at the final step, so the model's internal judgments about semantics and facts are not corrected.

Formal Grammars and Parsers: The "grammar" that constrained decoding relies on is typically described using JSON Schema or context-free grammar (CFG). The decoder internally maintains a state machine or incremental parser that advances its state with each accepted token and determines which tokens are legal next. Tools like Outlines, llama.cpp's GBNF, and XGrammar all follow this path. This also explains its capability boundary: it can express structural rules like "whether brackets are closed" or "whether an enum is legal," but it cannot express semantic constraints like "whether this city name matches the user's question."

The Shape Is Right, but the Values May Be Wrong

The problem lies at a layer that is easy to overlook: syntactic legality does not equal semantic correctness.

For example, suppose you ask the model to output a weather query tool call, and the Schema specifies two fields, city and unit, where unit only accepts "celsius" or "fahrenheit". Constrained decoding can guarantee that the output JSON definitely has these two fields and that unit is definitely one of these two values. But if the user asks "How many degrees is it in Beijing today?", the model could perfectly well output {"city": "Shanghai", "unit": "fahrenheit"}—flawless in format, completely off target in content.

The original article sums up this limitation precisely in one sentence: "The decoder can guarantee the shape of a tool call while leaving its values completely wrong."

The reason behind this is that constrained decoding only works at the boundary of token selection. It does not intervene in the model's understanding of context, does not check whether entity extraction is accurate, and does not verify whether numerical reasoning is correct. The mask is a syntactic filter, not a fact checker.

Syntactic Legality vs. Semantic Correctness: These are two orthogonal dimensions. Syntactic legality only requires that the output conform to the production rules of a given grammar; it is a purely formal judgment. Semantic correctness requires that the output match the real world, user intent, and contextual facts. A JSON string can simultaneously achieve "perfect structure" and "absurd content," just as in the example where a weather query for Beijing is filled in as Shanghai. Therefore, during evaluation, format compliance rate (such as JSON parsing success rate) and task accuracy must be measured separately; the former being close to 100% does not mean the latter is also high.

Typical Function Calling Failures: In tool calling scenarios, this kind of semantic error is especially common. The model may mistakenly fill in the unit as Fahrenheit instead of Celsius, parse the date as an adjacent day, replace an entity mentioned by the user with one more common in training, or incorrectly reuse the previous round's parameters in a multi-turn conversation. None of these errors will trigger parsing exceptions, but they will cause downstream APIs to return incorrect results or produce side effects, so additional parameter sanity checks and secondary confirmation of key fields are often required.

Why This Problem Is Easy to Underestimate in Engineering

The improvement in format compliance brought by constrained decoding is very intuitive, and it easily creates the illusion that "the problem has already been solved." Especially in the rapid prototyping stage, once developers see that JSON no longer throws parsing failures, they assume the entire pipeline is reliable.

But failure modes in production environments are often more subtle. Format errors immediately throw exceptions and are easy for monitoring to catch; semantic errors—wrong field values, confused entities, reversed units—flow quietly into downstream systems and may cause data contamination, incorrect API calls, or even bad business decisions.

A common misunderstanding is to think that constrained decoding can improve the model's reasoning quality. In fact, it does not change the model's internal representation; it only performs truncation at the output layer. The model's degree of understanding of the task and the precision of its grasp of user intent are completely unaffected by constrained decoding.

Practical Approaches in Real Use

Since constrained decoding cannot manage the correctness of values, where should it be placed in the toolchain?

Treat it as a format guarantee layer, not a quality guarantee layer. The responsibility of constrained decoding is to ensure that downstream parsers never need to handle malformed JSON. That value is significant in itself, but it cannot replace validation logic for output content.

On top of constrained decoding, validation rules still need to be designed for specific business scenarios. For example, perform field-level cross-validation for extraction tasks, check parameter ranges for tool calls, and perform secondary confirmation for key entities. These validations and constrained decoding are complementary, not substitutes.

Another direction worth noting is that the design of the Schema itself can carry part of the semantic constraints. JSON Schema features such as enum values, numeric ranges, and regex patterns also take effect in constrained decoding, and making reasonable use of them can move part of the semantic checking forward to the decoding stage. But this approach has clear boundaries—it can only express hard constraints like "the value belongs to a certain set," and cannot handle soft judgments like "whether the value fits the context."

Understanding the Boundary Is More Important Than Using the Tool

Constrained decoding is a mature and practical technology. It turns LLM structured output from "prayer-based programming" into reliable engineering practice. But every tool has its scope, and mistaking format guarantees for content guarantees is a costly cognitive bias in current LLM application development.

When building systems that depend on structured output, clearly distinguishing the two layers of requirements—"correct format" and "correct content"—and designing corresponding guarantee mechanisms for each is far more robust than simply relying on one decoding strategy.

One-line Summary

Constrained decoding only guarantees that LLM output format is legal; it cannot guarantee the semantic correctness of field values.

Paragraph Summary

Constrained decoding applies a syntactic mask during token sampling and sets the probabilities of illegal tokens to zero, making the output necessarily legal at the syntactic level, with format compliance rates approaching 100%, which is highly valuable for scenarios such as function calling and structured extraction. But the article points out a frequently overlooked boundary: syntactic legality does not equal semantic correctness. The decoder can guarantee the "shape" of a tool call while leaving field values completely wrong, such as filling a Beijing weather query with Shanghai or reversing the unit. These semantic errors do not trigger parsing exceptions and flow quietly into downstream systems, making them more insidious than format errors. Therefore, constrained decoding should be positioned as a format guarantee layer rather than a quality guarantee layer, and still needs to be paired with business logic such as field-level validation and parameter range checks; at the same time, some semantic constraints can be moved forward through Schema features such as enums, numeric ranges, and regex, but soft judgments that depend on context cannot be handled.