Valid JSON ≠ Contract Conformance: Five Free-Model Myths and a Pass-Rate Probe

At 2 a.m., a worker died. The payload parsed fine. At first, no one believed the alert.

That gap—parsing versus contract conformance—is exactly where many free-model pipelines quietly break. I keep seeing the same five claims in review discussions. Each one can be verified within an hour.

Myth One: If json.loads Passes, the Schema Was Followed

No. You have two gates, and most teams only install the first.

• Gate A—Parsing: Is this text actually a JSON value?
• Gate B—Contract: Does this value match your schema: required keys, types, enums, pattern matching, no extra fields?

A response can pass A and fail B in a dozen ways. {"ticket_id": 4411} parses cleanly but violates the ^TCK-\d{4,}$ pattern. An extra "notes" key also parses but violates additionalProperties: false.

JSON validity is a matter of syntax; contract conformance is a matter of contract. Only the latter is the trouble you actually face.

JSON Schema is the standard for describing and validating the structure of JSON documents (IETF draft). Its core keywords include type, required, properties, additionalProperties, enum, pattern, and so on. Gate B in this article is essentially a JSON Schema validation. What needs distinguishing is that json.loads only checks the syntax defined by RFC 8259, whereas schema validation checks semantic constraints. Also, constraints like additionalProperties: false have a common pitfall—by default, any field not listed in properties counts as extra. If the model returns fields common in its training data, such as "confidence" or "reason", it will be judged as failing. This kind of failure is often not a model capability issue but an overly strict contract design that needs to be aligned with the model's output habits.

Myth Two: If It Returns One Clean Object, the Model Is Reliable

A test suite that passes once is a demo, not a suite.

One call gives you n=1—an anecdote with braces. What you need is the pass rate and a confidence interval around it. Twenty trials let you put a defensible lower bound on conformance. One trial lets you tell a story at standup.

So ask the uncomfortable question: what's the lower bound, not the average?

The Wilson score interval is a commonly used confidence interval method for proportion estimation. Compared with the normal approximation (Wald interval), it performs more robustly with small samples and extreme proportions (close to 0 or 1), and does not produce intervals that go out of bounds or have zero width. The wilson function in the article's code implements this method. Understanding its meaning matters: when 17 of 20 trials pass, the point estimate is 0.85, but the 95% lower bound is about 0.62—meaning the true pass rate could be as low as 0.62. The decision table sorts by lower bound precisely because the lower bound is a conservative estimate of the "worst reasonable case," suitable for defending a release decision in review. If a team cares more about the Bayesian posterior, a Beta distribution can give a similar conclusion, but the Wilson interval requires no choice of prior and is better suited to quick probing.

Myth Three: Temperature 0 Gives You the Same Answer Twice

Test this against your own endpoint before you build on the assumption.

Even at temperature 0, outputs can drift across runs. Floating-point summation order, request batching, and silent backend version changes all move tokens. None of that is in your prompt.

Don't argue about it in the design doc. Send the same prompt twice, diff the raw strings, and write down what you see. "Deterministic enough to be a cache key" is a measurement, not a setting.

Temperature is a sampling parameter that scales the logits before applying softmax; lower temperatures make the distribution sharper. But temperature 0 usually does not equal a strict implementation of greedy decoding: many inference stacks still add very small sampling noise at temperature=0, or padding from other requests in a batch changes the numerical path. A deeper reason is the non-associativity of floating-point arithmetic—the same mathematical expression can produce different rounding results under different summation orders, which in turn flips the top-1 token. Moreover, servers may switch quantization versions, CUDA kernels, or speculative decoding strategies without notice. Therefore "determinism" should be treated as an end-to-end property that must be measured on the target endpoint, not inferred from parameters.

Myth Four: A Repair Loop Fixes Malformed Output for Free

A repair loop isn't free. It costs a second call, extra latency, and a second chance to fail in a different way.

Measure the marginal pass rate. If the first attempt conforms 70% of the time, how much does a second attempt add? If the answer is a few percentage points, you're buying noise with latency.

Set a stop rule before release:

1. First attempt, strict mode.
2. On failure, one repair call that includes the validation error verbatim.
3. On second failure, escalate. Don't loop—hand off to a human or a queue.

Three attempts is usually a bug disguised as a retry policy.

Myth Five: Free Access Means Zero-Cost Experimentation

Free access removes the bill, but not wall-clock time, rate limits, or your own attention.

That's the trap. A probe that burns forty minutes on rate-limited calls is expensive whether it's free or not. Budget probing like production work:

• Cap the number of trials. Twenty is enough for a first read.
• Sleep between calls. Politeness is cheaper than a 429.
• Read the rate-limit headers when the endpoint sends them. Don't guess.

The Probe: Two Gates, Twenty Trials, One Interval

Here's the harness. Standard library only, Python 3.10+ (for union types). It runs against any chat endpoint you can call.

I ran it through MonkeyCode's free model access on the free server option—the tool needs a machine that stays awake for twenty sequential calls. Disclosure: this article was prepared as part of a MonkeyCode product promotion.

# conformance_probe.py — measure conformance, don't trust a demo
import json
import math
import re
import time

TRIALS = 20
TICKET_RE = re.compile(r"^TCK-\d{4,}$")

PROMPT = (
    "Return one JSON object and nothing else. "
    'Keys: ticket_id (string, "TCK-" plus 4 or more digits), '
    "severity (one of: low, medium, high), "
    "summary (string, at least 10 characters). No extra keys."
)


def call_model(prompt: str) -> str:
    """Return raw assistant text. Wire this to your own client."""
    raise NotImplementedError("fill in your endpoint call")


def extract_json(text: str):
    fenced = re.search(r"```(?:json)?\s*(\{.*?\})\s*```", text, re.S)
    candidate = fenced.group(1) if fenced else text
    start, end = candidate.find("{"), candidate.rfind("}")
    if start == -1 or end <= start:
        return None, "no_object"
    try:
        return json.loads(candidate[start:end + 1]), None
    except json.JSONDecodeError as exc:
        return None, f"json_error:{exc.msg}"


def check(obj) -> str | None:
    """Gate B. Return a reason string, or None when conformant."""
    if not isinstance(obj, dict):
        return "not_an_object"
    extra = set(obj) - {"ticket_id", "severity", "summary"}
    if extra:
        return "extra_key:" + ",".join(sorted(extra))
    for key in ("ticket_id", "severity", "summary"):
        if key not in obj:
            return f"missing:{key}"
    if not isinstance(obj["ticket_id"], str) or not TICKET_RE.match(obj["ticket_id"]):
        return "bad:ticket_id"
    if obj["severity"] not in ("low", "medium", "high"):
        return "bad:severity"
    if not isinstance(obj["summary"], str) or len(obj["summary"]) < 10:
        return "bad:summary"
    return None


def wilson(passes: int, n: int, z: float = 1.96):
    if n == 0:
        return 0.0, 1.0
    p = passes / n
    denom = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / denom
    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / denom
    return max(0.0, centre - half), min(1.0, centre + half)


def run(trials: int = TRIALS):
    parsed = conformant = 0
    reasons: dict[str, int] = {}

    for _ in range(trials):
        raw = call_model(PROMPT)
        obj, err = extract_json(raw)
        if obj is None:
            reasons[err] = reasons.get(err, 0) + 1
        else:
            parsed += 1
            err = check(obj)
            if err is None:
                conformant += 1
            else:
                reasons[err] = reasons.get(err, 0) + 1
        time.sleep(1)  # be a good citizen; free endpoints still have limits
    lo, hi = wilson(conformant, trials)
    return {
        "trials": trials,
        "parse_rate": round(parsed / trials, 3),
        "conformance_rate": round(conformant / trials, 3),
        "lower_95": round(lo, 3),
        "upper_95": round(hi, 3),
        "failures": dict(sorted(reasons.items(), key=lambda kv: -kv[1])),
    }


if __name__ == "__main__":
    print(json.dumps(run(), indent=2))

What Each Part Actually Gives You

• extract_json distinguishes "wrapped in prose" from "not JSON at all." These are different bugs that need different fixes.
• check is Gate B. Hand-written for this schema; swap in jsonschema if you prefer, but pin the version.
• wilson gives you a lower bound you can defend in review, without assuming normality on a small sample.
• The failures histogram tells you which myth you're actually living. Most teams get one failure shape that recurs.

The regex r"```(?:json)?\s*(\{.*?\})\s*```" in extract_json uses non-greedy matching .*? and re.S (which makes . match newlines) to extract the first complete object from a Markdown code fence. But non-greedy matching with re.S can truncate prematurely when braces are nested inside the object—for example, {"a": {"b": 1}} would be cut off before the first }. The article then falls back to find/rfind, but this likewise cannot correctly handle cases where string literals contain braces. A safer approach in production is to use a scanner that understands JSON structure, or first require the model not to output fences. The simple implementation is kept here so that the probe relies only on the standard library; readers should adjust the extraction logic to their own output shapes.

Reading the Numbers

Sort by the lower bound, not the point estimate. The decision table:

Lower bound (95%)	Action
≥ 0.95	Strict mode, no repair loop, ship
0.80 – 0.95	One repair attempt, log every escalation
0.60 – 0.80	Narrow the schema. Fewer fields, simpler types
< 0.60	The model is unsuitable for this task. Retries won't save it

The last row matters most. Poor conformance is usually a task-shape problem, not a prompt problem. Adding retries to a badly scoped schema just spends more money on the same failures.

One more interpretation hint: if parse_rate is high but conformance_rate is low, your prompt is fine—your contract is unappealing. If parse_rate is low, stop reading and fix extraction first.

Limits and Who Should Skip This

This probe measures one schema, one prompt, one endpoint, one afternoon. It expires. Re-run it when the provider changes anything or when your schema gains fields.

Skip it entirely if you need byte-identical output across runs, or if the output feeds an audit trail. Free-tier models belong on classification, tagging, and draft paths—not on write paths you'll be asked to defend later.

Skip it, too, if your volume is five calls a day. At that scale, read the outputs yourself and save the tool for later.

And don't assume twenty trials is a guarantee. It's a first read with honest error bars, nothing more.

What I'd Keep

Replace "it worked" with "it conformed 17 out of 20 times, lower bound 0.62." That one sentence changes the whole design conversation.

Myths survive because no one runs the twenty lines that would kill them. So run them.

If you want to try this before wiring up billing, MonkeyCode's free model access plus the free server option is a starting point—bring your own schema and your own skepticism.

One-line Summary

Use twenty trials and a Wilson lower bound to separate JSON parsing from contract conformance and bust five free-model myths.

Paragraph Summary

The article points out that free-model pipelines commonly mistake "JSON parses" for "conforms to the contract," and distills this into five myths verifiable within an hour: json.loads passing isn't schema matching, one clean output isn't reliability, temperature 0 doesn't guarantee determinism, repair loops aren't free, and free access isn't zero cost. The author provides a standard-library-only probe script that runs twenty trials against any chat endpoint, separately measuring parse rate and conformance rate, and gives a 95% lower bound via the Wilson interval. Decisions should be made by the lower bound rather than the average: ≥0.95 supports a strict-mode release, 0.60–0.80 means narrowing the schema, and below 0.60 means the model is unsuitable for the task. The article also emphasizes that the failure histogram reveals specific failure shapes and warns that probe results expire, requiring re-runs after schema or endpoint changes.