JSON Schema Passes Validation, But the Agent Output Is Empty?

Title

JSON Schema Passes Validation, But the Agent Output Is Empty?

Meta Description

JSON Schema validation only checks data shape, not empty Agent output. This article presents seven measurable signals of "emptiness" to intercept VALID_BUT_EMPTY false greens in a pipeline.

Article Content

When the Agent was asked to find five flaws in a contract, it returned five findings. The JSON was perfectly legal: five objects, all required fields present, severity levels from the allowed enum, no extra keys. jsonschema.validate returned None—no exception, no error path. The pipeline marked this step as a green pass.

But look closely at those five findings: they are completely identical.

The Agent did nothing, yet reported success in a perfectly legal format. This is not a rare failure mode—it is the most common one, because it is the only failure that leaves no trace in the "shape" of the data.

Form Is Not Content

Schema validation answers only one question: does the shape of this data conform to our contract? Types, required fields, enums, formats, nesting—every check is about the envelope, none about the letter inside.

This is not a flaw of JSON Schema. Schema was never asked "is this meaningful?" We validated what the artifact is—a data structure—and then quietly treated "legal" as "good." These are two different assertions, and the gap between them is exactly where agents live.

JSON Schema itself is a vocabulary for describing and validating the structure of JSON documents, standardized by the IETF across multiple RFCs (the core specification is a companion draft to RFC 8259, with the current version being 2020-12). It supports keywords such as type, properties, required, enum, pattern, and minimum, which precisely constrain data types, required-ness, value ranges, and nesting. But Schema's design goal has always been "interoperability"—getting different systems to agree on the shape of the same data, not judging whether the data holds up in a business sense. For example, {"status": "ok"} and {"status": "ok", "detail": ""} may both conform to the Schema, but the latter may be semantically incomplete. Understanding this is the prerequisite for reading this article: Schema is an envelope specification, not a letter content review.

What "Empty" Actually Looks Like

Missing output is easy to handle: there is nothing to validate, the response fails, the error is loud. The interesting cases are all "presence without information"—the artifact is fully formed but says nothing:

• Five list items are completely identical—the "five findings" are actually one finding copied five times;
• "message": "TODO"—the field exists because the Schema requires it, not because anyone filled it in;
• "recommendations": []—a Schema that allows empty arrays reports success on an empty array;
• A required field is carefully left at the Schema's default value;
• A summary sentence stops halfway because the model hit the token limit and never mentioned it;
• A 40-word paragraph has only 9 unique words.

Each of these passes validation. If the reviewer is another LLM taking a quick look at "does it look right," most will pass that review too.

Seven Measurable Signals

The key move is to reject unfalsifiable checks. "This response feels meaningless" is not a test—you cannot show it to someone and argue about the result. But "this required field consists entirely of the word example" is a fact about bytes.

So we measure emptiness where it can be measured—seven signals, each checkable and contestable:

1. Uniform list items (uniform_list): if more than half the items in a list are identical (≥90% is severe), the list is a placeholder for a list. A single repetition may be honest; mostly repetition is an absence of work.
2. Placeholder values (stub_value): TODO, TBD, placeholder, example, test, lorem ipsum, n/a—including their Chinese equivalents. Severity is high when the entire field is a placeholder, low when placeholder words appear within real text.
3. Blank strings and empty containers (blank_string, empty_list, empty_object): pure whitespace strings and []/{} satisfy the Schema but satisfy nothing else.
4. Truncation markers (truncation): output ends with ..., [truncated], continued—the model was cut off without saying so.
5. Low-diversity repetition (repetitive_text): for text of 12+ words, if fewer than about one-third are unique, it is filler.
6. Schema default left unchanged (schema_default): a required field holds exactly the Schema's own default, meaning nobody filled it in.
7. Enum field stuck on the first value (enum_first_value): a categorical field always holding the first enum option is a guess disguised as a decision.

None of these check semantics. They check the mechanical fingerprints of absence—and because each fingerprint is a measurement, each can be shown to a reviewer: here is the field, here is the value, here is why it fired.

Among the signals above, uniform_list and repetitive_text both rely on the statistic of "repetition degree," but they detect at different granularities: the former targets element-level duplication in structured arrays, the latter targets word-level diversity in natural language text. Word-level diversity is usually measured by the unique token ratio, i.e., the number of unique words divided by the total number of words. For short texts, this ratio is naturally high; for texts of 12+ words, if the ratio is below about one-third, it indicates that a large number of words are being repeated and padded, commonly seen when a model falls into a loop or parrot mode. Similarly, the schema_default signal works because the Schema's default keyword is intended to be a "suggested value" rather than a "required value." When a required field happens to equal the default, it is highly likely that the generation side did not actually fill it in, but rather let the serialization library or template auto-complete it.

Verdict, Not Feeling

Signals aggregate into three verdicts:

• CONTENT_PRESENT: none of the above signals fired.
• THIN_CONTENT: some moderate signals—worth a look.
• VALID_BUT_EMPTY: the form is fine, the content is suspect.

The third name is deliberate. It means "form is fine, content is suspect"—and not "content is missing." Suspicion needs a human; a verdict needs evidence. The tool presents both and pretends to be neither.

There is also an honest limitation, written in the README because it matters: the absence of signals does not prove meaningfulness. It only means a coarse check did not fire. A response can be vague, circular, and useless in seven ways we did not think to measure. Passing is a statement about the check, not a certificate about the content.

Where It Belongs in the Pipeline

This check belongs at the boundary where Agent output stops being a "reply" and starts being a "result"—before it is paid for, forwarded, merged, or shown to a customer as completed work.

Two properties matter there. First, it must run before the consequences—discovering five identical findings after the refund has been processed is an incident report, not a guardrail. Second, the verdict must be evidence-shaped: if you are going to reject an Agent's output, you owe it a specific, contestable reason—uniform_list: 5 of 5 items are identical—not a feeling.

You cannot retry what you cannot detect. An empty artifact that passes validation is a false green, and false greens are the only failure mode that survives every dashboard.

If you want to see it run: attest checks artifacts at four layers—form and emptiness (this article), prompt-injection shape, provenance signatures, and permissions—before any downstream trusts them. The code is open source, the demo is just a form, and the emptiness layer is forty-plus tests corresponding exactly to the signals above.

One-line Summary

JSON Schema validates only data shape and cannot detect "legal but hollow" Agent output such as duplicate lists or placeholders; seven measurable signals are needed to intercept false greens before consequences occur.

Paragraph Summary

The article points out that Agent output passing JSON Schema validation yet being content-empty is a common failure: five completely identical findings, TODO placeholders, empty arrays, truncated text, and more can all legally pass. Schema only checks data shape, not semantics, causing "legal" to be mistaken for "good." To address this, the author proposes seven mechanically measurable signals—identical list items, placeholder values, blank containers, truncation markers, low-diversity text, unchanged Schema defaults, and enums stuck on the first value—each corresponding to displayable evidence. The signals aggregate into three verdicts: CONTENT_PRESENT, THIN_CONTENT, and VALID_BUT_EMPTY, emphasizing that verdicts must be based on evidence rather than feelings, should run before consequences such as payment or merging occur, and acknowledging that passing the check does not equal meaningful content.