CaseHOLD
A benchmark in the legal domain used to evaluate model performance on judicial reasoning tasks, typically involving the selection of the correct legal basis given a holding of a judgment.
Timeline (last 90 days)
This counterfactual audit covers seven open-weight models (8B to 70B parameter scale) and four benchmarks spanning judicial reasoning and contract reasoning.
After the legal basis cited by the model is replaced with an irrelevant authority, the rate at which the decision changes is only 0.0%–21.7% on CaseHOLD, 30.0%–76.7% on ECHR and SCOTUS, and 43.3%–50.0% on ContractNLI.
All Facts (2)
This counterfactual audit covers seven open-weight models (8B to 70B parameter scale) and four benchmarks spanning judicial reasoning and contract reasoning.
50%UnverifiedAfter the legal basis cited by the model is replaced with an irrelevant authority, the rate at which the decision changes is only 0.0%–21.7% on CaseHOLD, 30.0%–76.7% on ECHR and SCOTUS, and 43.3%–50.0% on ContractNLI.
50%