Using Stability Scores to Decide Whether to Trust Model-Generated Documents

Reading an AI-generated document draft once can easily persuade you with its fluent, specific phrasing, but that confidence may mask the fact that the model's output is inconsistent across different samples. A new approach proposes: instead of asking "is this draft good?", ask "does it converge?". By sampling the same paragraph prompt multiple times, computing wording similarity and detecting factual conflicts, you can assign each document paragraph an explicit author role—model draft, model outline, or human-written.

The Blind Spot of Single-Sample Judgment

A model-generated document reads like a decision because it is coherent, specific, and complete. But run the same prompt again, and you will often get different numbers, different ordering, another account of what happens on failure. That variance is information about the input, not noise to be averaged away. A paragraph that changes shape across samples usually means the underlying spec, configuration, or error text is ambiguous or missing.

So the useful question is not "is this draft good?" but "does this draft converge?". Convergence is cheap to measure, hard to fake, and directly actionable for ownership.

Core Rule: k Samples, Two Signals, Three Verdicts

Collect at least three samples per paragraph, and five if you can afford it. Compute the average pairwise similarity of normalized token shingles, then separately detect claim-type tokens (numbers, sizes, durations, URLs) that appear in some samples but not all. Similarity tells you whether the wording has settled; the divergent-claim check tells you whether the facts have settled.

Average similarity	Divergent claims	Verdict
Any	Yes	HUMAN—inputs contradict each other, the document cannot stabilize
≥ 0.90	No	MODEL-DRAFT / HUMAN-APPROVE—review tone and correctness
0.75 – 0.89	No	MODEL-OUTLINE / HUMAN-NORMATIVE—model proposes structure, human writes the binding statements
< 0.75	No	HUMAN—input material may be ambiguous

Divergent claims deliberately override the score, because a paragraph that confidently cites three retry counts is worse than one that cites none.

Jaccard similarity and shingles: A shingle is a fragment formed by splitting text into consecutive k tokens (usually words or characters). For example, "model output unstable" with k=2 produces two shingles: "model output" and "output unstable". Jaccard similarity is defined as the size of the intersection of two sets divided by the size of their union, ranging from 0 to 1. In this method, each sample's token sequence is first normalized (e.g., lowercased, punctuation removed) to generate a shingle set, then the pairwise Jaccard similarity across all samples is computed and averaged. This metric is sensitive to wording changes but somewhat tolerant of word-order adjustments, making it suitable for measuring "whether the phrasing is consistent".

Divergent claim detection: Claim-type tokens are fragments that carry factual commitments once they appear, typically including numbers, time units (ms, s), percentages, URLs, version numbers, etc. The detection logic is: extract the set of claim-type tokens from each sample; if a token appears in some samples but not all, flag it as a divergent claim. The method deliberately does not judge which claim is correct, only whether claims are stable. Even if a paragraph has highly similar wording, the presence of divergent claims indicates an unresolved conflict in the input material.

Offline Test Framework: Prove the Tool First, Then Trust the Model

The provided Python script draft_stability.py contains no network code, so the same file can run in CI, on a laptop, and in scheduled jobs. It computes the overlap of shingle sets via Jaccard similarity and uses regular expressions to capture claim-type tokens such as numbers, time units, percentages, and URLs.

A well-designed fixture example proves the entire scoring path. samples.json contains two paragraphs: the install paragraph's three samples are highly consistent in wording and receive a MODEL-DRAFT / HUMAN-APPROVE verdict; the retry-behavior paragraph's three samples claim 3, 5, and 2 retries respectively, with backoff times ranging from 200ms to 100ms to 250ms, triggering the divergent-claim check and being directly judged as HUMAN.

Running python draft_stability.py --fixture samples.json --k 3 --out record.md, the second line of the output is the entire method in miniature: the model can fluently write about retry behavior, but the retry policy was never decided anywhere the model could read it.

Fixture mode: In software testing, a fixture is a set of pre-constructed, reusable test data and preconditions. The draft_stability.py in this article loads a local JSON file via the --fixture parameter, where each paragraph contains multiple pre-generated sample texts. This way, the scoring logic (similarity computation, claim detection, verdict mapping) can be independently verified without depending on any model endpoint. The core value of fixture mode is decoupling "is the tool correct" from "is the model available": CI pipelines can always run fixture tests to ensure the scoring path has no regressions, while live sampling tests run as a supplement when the endpoint is reachable. If the live endpoint is unreachable, fixture tests should still execute and correctly report failure rather than silently skipping.

Wiring in Live Sampling: A One-Function Contract

The test framework deliberately knows nothing about your client. When wiring in your own client, keep the contract to one function: takes a prompt string, returns generated text. Then collect 5 samples with a list comprehension, write them to live.json, and run with --k 5. This way record.md records the input you actually used and the number of samples.

Turning Verdicts into Task Assignments

MODEL-DRAFT means an editor reviews tone and a reviewer checks commands; MODEL-OUTLINE means a human must write every sentence that states a limit, guarantee, or default; HUMAN means the next task is not writing at all—find the config file, incident record, or owner that resolves the question, and only then return to the prompt.

Stability is a property of a moment, not a permanent label. When the config, error strings, or release notes change, rerun the test framework and treat a dropped score as a signal that the paragraph's source of truth has drifted. Commit record.md alongside the page so reviewers can see which paragraphs were measured and which were merely written.

Stability as a property of a moment: Convergence measures "under current input conditions, do multiple model samples point to the same answer", so it reflects the determinacy of the input specification, not an inherent property of model capability. When the underlying config, error strings, release notes, or any context the model reads change, a previously high score may drop quickly. Committing record.md alongside the documentation page is equivalent to attaching a "measurement snapshot" to each paragraph, allowing reviewers to distinguish "measured paragraphs" from "merely written paragraphs". A dropped score should be treated as an alert that the input truth has drifted, not as evidence of model degradation.

Limitations and Scope

Similarity measures wording, not truth: five samples can unanimously agree on a wrong retry count and get a beautiful score, which is why the divergent-claim check exists and why humans still review MODEL-DRAFT. Deterministic samplers inflate every score, so sample with the variants that exist in what you actually ship. Very short paragraphs degrade to a single shingle, changelogs and generated examples are inherently unstable, and legal or contract text should never be assigned by score.

This tool does not measure model quality; it measures whether your input has settled. For loops that need to generate the same prompt many times, free model access can lower sampling cost, but the test framework's fixture mode should always remain in CI as a baseline—when the live endpoint is unreachable, the check must still execute and correctly fail.

One-line Summary

Compute convergence through multiple samples to classify AI document paragraphs as model draft, model outline, or human-written.

Paragraph Summary

The article points out that reading an AI-generated document once is easily misled by fluency, while sampling the same prompt multiple times can reveal whether the output converges. The method collects 3 to 5 samples per paragraph, computes the average pairwise Jaccard similarity of normalized token shingles, and detects claim-type tokens (numbers, times, URLs, etc.) that appear in some samples. Based on similarity thresholds (0.90 and 0.75) and divergent-claim detection, paragraphs are judged as MODEL-DRAFT, MODEL-OUTLINE, or HUMAN. Divergent claims take priority over similarity scores. The provided offline Python framework first validates the scoring path with a fixture, then wires in live sampling through a single-function contract. Verdicts translate directly into editing tasks: review tone, have a human write binding statements, or trace the input source. Stability is a property of a moment and must be rerun when inputs change; the method measures whether input has settled, not model quality.