Schema Validation
An engineering practice that verifies the structure and field validity of structured output returned by a model, ensuring that fields such as coordinates, areas, and labels satisfy expected constraints.
Timeline (last 90 days)
Schema validation can only guarantee structural compliance and cannot determine whether field contents are fabricated, so it must be used together with semantic-level checks such as AST parsing and dry-run interpreters.
Because models cannot fundamentally distinguish "data" from "instructions", prompt-based protection alone is unreliable; a viable approach is to have the model only output suggestions, with all actions that have side effects executed by application code according to a whitelist, and to perform schema validation and permission checks on model output.
Over-reliance on LLM-as-Judge leads to a slow and costly test suite, and high coverage of bottom-layer schema tests is the foundation for fast feedback.
Bottom-layer schema tests run in milliseconds, while top-layer LLM-as-Judge evaluations cost only a few cents per API call.
When a language model computes a numerical average, running it twice in succession may yield different results (e.g., 49.6 and 49.8), both of which can pass schema validation but may not be correct.
Schema validation only verifies whether the output format is valid and cannot guarantee the correctness of numerical values.
Strict schema validation creates manual review workload but prevents cleanup jobs from inventing policies at write boundaries, which is a trade-off between throughput and control.
All Facts (7)
Schema validation can only guarantee structural compliance and cannot determine whether field contents are fabricated, so it must be used together with semantic-level checks such as AST parsing and dry-run interpreters.
50%UnverifiedBecause models cannot fundamentally distinguish "data" from "instructions", prompt-based protection alone is unreliable; a viable approach is to have the model only output suggestions, with all actions that have side effects executed by application code according to a whitelist, and to perform schema validation and permission checks on model output.
50%UnverifiedOver-reliance on LLM-as-Judge leads to a slow and costly test suite, and high coverage of bottom-layer schema tests is the foundation for fast feedback.
50%UnverifiedBottom-layer schema tests run in milliseconds, while top-layer LLM-as-Judge evaluations cost only a few cents per API call.
50%UnverifiedWhen a language model computes a numerical average, running it twice in succession may yield different results (e.g., 49.6 and 49.8), both of which can pass schema validation but may not be correct.
50%UnverifiedSchema validation only verifies whether the output format is valid and cannot guarantee the correctness of numerical values.
50%UnverifiedStrict schema validation creates manual review workload but prevents cleanup jobs from inventing policies at write boundaries, which is a trade-off between throughput and control.
50%