ContractNLI
A benchmark dataset designed for contract natural language inference. It contains 550 non-disclosure agreements (NDAs) and 17 hypothesis statements, requiring models to determine entailment, contradiction, or neutral relationships between a contract and a hypothesis, and to annotate supporting evidence spans. It features a two-level annotation structure covering conclusion labels and paragraph-level evidence.
Timeline (last 90 days)
This counterfactual audit covers seven open-weight models (8B to 70B parameter scale) and four benchmarks spanning judicial reasoning and contract reasoning.
After the legal basis cited by the model is replaced with an irrelevant authority, the rate at which the decision changes is only 0.0%–21.7% on CaseHOLD, 30.0%–76.7% on ECHR and SCOTUS, and 43.3%–50.0% on ContractNLI.
The study uses the ContractNLI dataset, in which manually annotated evidence spans are withheld entirely until the evaluation stage and are not visible to the model during training.
The ContractNLI dataset contains 550 non-disclosure agreements (NDAs) and 17 hypotheses, requiring models to determine entailment, contradiction, or neutrality and to annotate the evidence spans supporting the verdict.
Aplomb 1 的训练数据包含 WinoGrande 和 ContractNLI 这两个基准的公开训练集，而它们也在 38 项索引基准之中
The researcher constructed an evaluation benchmark by injecting contradictions into the corpus, using three types of data: university HR handbooks, synthetic corpora with injected contradictions, and ContractNLI with external annotations.
ContractNLI is a dataset built by Stanford for NLI tasks that contains real contradictions; its task format is hypothesis-to-clause rather than the symmetric clause-to-clause comparison.
All Facts (7)
This counterfactual audit covers seven open-weight models (8B to 70B parameter scale) and four benchmarks spanning judicial reasoning and contract reasoning.
50%UnverifiedAfter the legal basis cited by the model is replaced with an irrelevant authority, the rate at which the decision changes is only 0.0%–21.7% on CaseHOLD, 30.0%–76.7% on ECHR and SCOTUS, and 43.3%–50.0% on ContractNLI.
50%UnverifiedThe study uses the ContractNLI dataset, in which manually annotated evidence spans are withheld entirely until the evaluation stage and are not visible to the model during training.
50%UnverifiedThe ContractNLI dataset contains 550 non-disclosure agreements (NDAs) and 17 hypotheses, requiring models to determine entailment, contradiction, or neutrality and to annotate the evidence spans supporting the verdict.
50%UnverifiedAplomb 1 的训练数据包含 WinoGrande 和 ContractNLI 这两个基准的公开训练集，而它们也在 38 项索引基准之中
50%UnverifiedThe researcher constructed an evaluation benchmark by injecting contradictions into the corpus, using three types of data: university HR handbooks, synthetic corpora with injected contradictions, and ContractNLI with external annotations.
50%UnverifiedContractNLI is a dataset built by Stanford for NLI tasks that contains real contradictions; its task format is hypothesis-to-clause rather than the symmetric clause-to-clause comparison.
50%