No Evidence Annotation Needed: Training on Verdict Labels Alone Can Recover Legal Text Evidence

The Overlooked Reality: Verdicts Are Kept, Evidence Is Not

In many review workflows — contract review, compliance determination, legal adjudication — what typically gets preserved is the conclusion alone. The key passages that support that conclusion, the specific clauses cited, are rarely annotated separately. The reason is straightforward: labeling those evidence spans costs far more than simply recording a verdict.

This raises an interesting question: if training data contains only "verdicts" with no "evidence annotations," how well can a small language model reconstruct the underlying evidence on its own? A newly published arXiv paper (arXiv:2610.06962) tackles exactly this question. Researchers tested whether models trained solely on final verdicts — with no human evidence labels used at any point during training — can reconstruct verifiable citation evidence.

Accuracy Does Not Equal Correct Evidence

The study used the ContractNLI dataset, with human-annotated evidence spans withheld entirely until the evaluation stage — invisible to the model throughout training. This design reveals a core insight: matching the correct verdict and actually citing the correct evidence spans are two different things.

The paper compared six systems and found only weak correlation between these two scores, with inconsistent rankings across systems. In other words, a model that leads on verdict accuracy is not necessarily better at evidence localization (span F1). This matters because when citations must be "verifiable" — as in legal or compliance contexts — using accuracy alone to evaluate model quality is misleading.

The practical implication is direct: if your system needs to show a human reviewer "this is the clause I'm relying on," looking only at whether the conclusion is correct is nowhere near sufficient.

---

ContractNLI is a benchmark dataset designed specifically for contract natural language inference. It contains 550 non-disclosure agreements (NDAs) and 17 hypothesis statements. Each sample requires the model to determine whether the contract "Entails," "Contradicts," or does "Not Mention" a given hypothesis, while also identifying the specific evidence spans that support that determination. This dual-layer annotation structure — verdict labels plus paragraph-level evidence — makes it an ideal testbed for examining whether conclusion accuracy and evidence localization ability can be decoupled. Span F1 measures the overlap between the model's predicted evidence spans and the human-annotated spans, while verbatim citation requires the model's quoted text to match the source document exactly — a stricter verifiability standard.

---

Two Approaches: Label-Only Training vs. Rejection Sampling

The study compared two post-training methods that require no evidence annotations.

Label-Only Training

This approach uses bare verdict labels directly as the supervision signal. It achieved 0.896 accuracy and 0.564 span F1. On the verbatim citation metric, it improved from 0.597 before training to 0.729. Notably, even without any evidence annotations, the model managed to "incidentally" learn to locate more accurate citation spans while training on verdicts.

Rejection Sampling

Rejection sampling takes a different approach: it prompts the model to generate multiple reasoning traces, retains only those where the verdict matches the record, and then uses an automatic "source-grounding score" to select the best trace. This method achieved 0.797 accuracy and 0.556 span F1, both up from the pre-training baseline of 0.747 and 0.493 respectively. Verbatim citation improved from 0.597 to 0.701.

Comparing the two: label-only training is clearly stronger on accuracy, while on evidence localization and verbatim citation the two methods are close, each with a slight edge in different respects.

---

Rejection Sampling Fine-Tuning (RFT) is a post-training paradigm that requires no human preference annotations, popularized by Meta in the LLaMA 2 research. The core idea is to sample multiple outputs for a given input, filter out incorrect ones using an external criterion (here, whether the verdict is correct), and then fine-tune on the remaining "successful traces." Compared to RLHF, it requires neither a separate reward model nor human preference pairs, making it considerably cheaper to implement. This paper adds an additional filtering step using a "source-grounding score" — among samples with correct verdicts, it selects the trace whose citations best align with the original contract text. This implicitly signals to the model which citation style is more trustworthy, without any human evidence annotations.

---

What the Numbers Mean

Putting the key figures together makes the picture clearer:

Metric	Pre-Training	Label-Only	Rejection Sampling
Accuracy	0.747	0.896	0.797
Span F1	0.493	0.564	0.556
Verbatim Citation	0.597	0.729	0.701

Both methods improved evidence recovery without any human-annotated evidence. This is the paper's most valuable finding: improving evidence recovery does not necessarily require expensive human annotation investment — the verdict labels themselves contain enough supervisory signal for the model to extract.

That said, the authors are admirably restrained. They explicitly note that experiments using a single random seed on a single corpus cannot establish which method is superior. It's an honest acknowledgment of limitations — both methods work, but determining which is better requires broader validation.

Implications for Practitioners

This research has several practical takeaways for teams building interpretable AI systems.

First, your choice of evaluation metrics determines what you optimize for. If the end product requires verifiable citations, evidence metrics like span F1 and verbatim citation should be part of your evaluation suite — not just verdict accuracy.

Second, expensive evidence annotation may not be necessary. In annotation-budget-constrained settings, post-training on existing verdict records already yields meaningful evidence recovery gains. This is particularly appealing for organizations that have accumulated large volumes of historical decision data but lack paragraph-level annotations.

Third, method selection requires trade-offs based on your goals. If verdict accuracy is the priority, label-only training has the edge. If you care more about controllability in the generation process and source grounding, rejection sampling provides a scoring-based filtering mechanism.

This is a modest paper, but it addresses a long-underappreciated engineering reality: many organizations hold data that captures only "conclusions," and AI can help recover the "reasoning" that was lost along the way.